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. Introduction. 

in 

This is the first of two companion papers in which we try to generalize the results 
of one of us, [19], to higher dimensions. In that paper, it was established that the 
nonsingular Fourier coefficients of the derivative at of certain incoherent Eisen- 
stein series on the metaplectic group in 4 variables are closely related to the value 
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of the height pairing of a pair of arithmetic cycles on a Shimura curve. Both sides 



of the identity to be proved turned out to be a sum of terms enumerated by the 
places of Q . In the present papers we are only concerned with the situation at a 



nonarchimedean prime. 

It is a hope, already expressed in [19], that a similar relation holds in general be- 
tween the value of the derivative at of certain incoherent Eisenstein series on the 



metaplectic group in 2n variables and the height pairing of suitable arithmetic 
cycles on Shimura varieties associated to orthogonal groups of signature (n — 1, 2) . 
This would constitute an arithmetic analogue of the result of the first author [18] 
which relates the value at 1/2 of certain coherent Eisenstein series with the inter- 
section pairing on suitable classical cycles on these Shimura varieties. 
A first difficulty with this general program is that models over the integers of the 
Shimura varieties associated to orthogonal groups are not well understood. For low 
values of n there are, however, exceptional isomorphisms which relate the groups 
in question to symplectic groups, and the Shimura varieties associated to them have 
integral models which one can investigate. In the present paper we are concerned 
with the exceptional isomorphism which relates the orthogonal group of signature 
(3, 2) with the symplectic group in 4 variables. In the companion paper [21] we are 
concerned with the Shimura variety associated to an orthogonal group of signature 
(2, 2) which is related to certain Hilbert-Blumenthal surfaces. 
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Let us now be more specific about the contents of this paper. Let B be an 
indefinite quaternion algebra over Q . Let C = M 2 (B) and put 



where x i— > x' = t x L is the involution on C induced by the main involution on B . 
Then (V, q) , with q defined by x 2 = q(x) • 1 , is a quadratic space of signature 
(3, 2) and the group G = GSpin(V) of V can be identified with a twisted form of 
the group of symplectic similitudes in 4 variables. Let T> be the space of oriented 
negative 2-planes in V(M) and let K be a compact open subgroup of G(Af) . 
Then, the Shimura variety Sh(G, V)k , whose complex points are given by 



is a (twisted) version of the Siegel 3-fold over Q . For example, the case of the split 
quaternion algebra B = M 2 (Q) yields the usual Siegel modular variety of genus 
2. 

The exceptional isomorphism of G = GSpin(V) with a form of GSp^ plays a 
fundamental role throughout the paper. In particular, we use it to construct a 
good integral model of Sh(G, T>)k ■ More precisely, we fix a prime p > 2 such that 
B is unramified at p and take K of the form K = K P .K P , where K p C G(A P j) 
is sufficiently small and where K p is the natural maximal compact open subgroup 
of G(Q P ) . Then we use the modular interpretation of Sh(G, T>)k to construct a 
smooth model M. over Spec Z( p ) , as a parameter space of certain abelian varieties 
with additional structure. 

Algebraic cycles on Sh(G, V) K were defined analytically in [18] as follows. For 
x G V n let q(x) = \ ((xi, Xj)) G Sym n (Q) , be the matrix of inner products of the 
compnents of x for the symmetric bilinear form ( , ) associated to q . Assume 
that d = q(x) is positive-definite (hence n < 3), and let V x be the subspace of 
oriented negative 2-planes orthogonal to all entries of x . Let G x be the pointwise 
stabilizer of x . Then Sh(G x ,V x ) is a sub-Shimura variety of Sh(G, V) , and thus 
defines a cycle of codimension n in Sh(G, T>)k ■ These cycles are a special case 
of the totally geodesic cycles in locally symmetric spaces studied in [20] and else- 
where. A slight generalization of the previous construction yields a cycle Z(d,oj; K) 
of Sh(G, T>)k which is associated to any positive definite d G Sym n (Q) and any 
K -invariant compact open subset lo of V(Af) n . 

The next step is to give a modular definition of these cycles. First, for one of the 
abelian varieties parametrized by M. , we define the notion of a special endomor- 
phism (Definition 2.1). The space of such endomorphisms is a finitely generated free 
Z -module equipped with a quadratic form q . The cycle Z(d, uj; K) ( = Z(d, u) if 
K is fixed) is then obtained by imposing an n -tuple j of special endomorphisms 
such that g(j) = d , and satisfying an additional compatibility with respect to u . 



(0.1) 



V = {x G C | x' = x, tr°(x) = 0} , 



(0.2) 



Sh(G,V) K (C) = G(Q) \ [V x G(Af)/K], 
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If ui satisfies an integrality condition at p , this definition can be used to extend 
the cycle Z(d, uj) to a cycle Z(d, uj) for the integral model M. of the Shimura va- 
riety. Here, by a cycle on M. , we mean a scheme which maps by a finite unramified 
morphism to M. . At this point we meet a very important problem: in contrast to 
M. , the cycles Z(d,u>) will no longer be smooth, in general. In fact, they often are 
not flat over Z( p ) and may even have embedded components. Our justification for 
our choice of this integral extension of the classical cycles is that their definition is 
very simple, has a nice inductive structure with respect to intersection, and that we 
are able to prove something about them. Before stating these results, we note that, 
while the arithmetic cycles Z(d, uj) can be defined for any d G Sym n (Q) , any n , 
they are nonempty only when d is positive semidefinite and with coefficients in 
Z(p) . 

We fix positive integers ni, . . . , n r with n\ + . . . + n r = 4 and, for each i , 
we choose a positive definite di G Sym n . (Z( p )) and a K -invariant open compact 
subset Ui G V(A) ni . The resulting cycles Z(di,u>i) on M. have generic fibres of 
codimension rii . We form the fibre product 

(0.3) Z = Z(di, u>i) Xm . . . Xm Z(d r , u r ) . 

To each point £ of Z , we then associate its fundamental matrix Tg G Sym 4 (Z( p ))>o , 
defined by T% = q(j) where j = (ji,...,j r ) is the 4-tuple of special endomor- 
phisms imposed at a point of the fiber product. Note that the diagonal blocks of T 
are (di, . . . , d r ) . The function £ is locally constant for the Zariski topology 

on Z and induces a disjoint sum decomposition in which the summands are again 
special cycles of a definite kind, 

(0.4) Z= ]J Z(T,u) . 

T G Sym 4 (Z (p) )> 
diag(T) = (d 1 ,...,d r ) 

Here uj = ui\ x . . . x ui r . The summand on the right corresponding to T is the set 
of points £ where = T . This decomposition illustrates the inductive nature of 
the special cycles mentioned above. 

The decomposition (0.4) bears some formal similarity to the partitioning into 
isogeny classes that occurs in the approach of Langlands-Kottwitz to the calcu- 
lation of the zeta function of a Shimura variety. In that approach the stable conju- 
gacy class of the Frobenius endomorphism is the most basic invariant of an isogeny 
class. In our context this role is played by the fundamental matrix. One of our 
discoveries is that the fundamental matrix and more specifically its divisibility by 
p governs the intersection behaviour of the special cycles. Thus if det(T^) ^ , 
i.e. T = is positive definite, then the point £ lies in characteristic p and is not 
the specialization of a point of Z in characteristic 0. In this case, the connected 
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component Z(T, u) of Z containing £ consists entirely of supersingular points of 
M. . Contrary to what one might expect, however, the condition det(T^) 7^ is 
not sufficient to ensure that £ is an isolated point of intersection. One of our main 
results is the characterization of when this is the case. 

Theorem 0.1. Let ^ 6 Z with det(T^) 7^ . Then £ is an isolated intersection 
point if and only if represents 1 over 7L V . In this case the underlying abelian 
variety is isomorphic to a power of a supersingular elliptic curve. 

When T = T{. does not represent 1 over Z p (but still is positive definite), then 
the connected component Z(T, ui) of Z containing £ is a union of projective lines 
and, in fact, one can enumerate these lines. It turns out that the more divisible 
is by p , the more components there will be. A more thorough analysis of the set of 
irreducible components can be found in [21]. We point out that this phenomenon 
of excess intersection does not occur in the case of Shimura curves at a place of 
good reduction [19], but it does occur at a place of bad reduction [22]. 
With the previous notation let us put 

(0.5) <Z(d 1 ,u 1 ),...,Z(d r ,u r )>^= e ^ > 

tez, 

£ isolated 

where each isolated intersection point £ appears with multiplicity e(£) = \gOz,£ , 
the length of the local ring of Z at £ . 

We next come to the relation with Eisenstein series, for which we refer the reader 
to section 8 or the first part of [19] for more details. Let W be a symplectic space 
over Q of dimension 8 and let 

(0.6) W = Wi + . . . + W r 

be a decomposition of W into symplectic spaces Wi of dimension 2ni . Let 
(0.7) % : Mpi, A x ... x Mp rA -> Mp A 

be the corresponding embedding of metaplectic groups. Let <fr(s) be the stan- 
dard section of the induced representation I(s, xv) of Mp& which is of the form 
$(s) = $00(5) <8> <&/(s) • Here the finite part is associated to the Schwartz function 
char a; G S(V(Af) 4 ) under the natural map S(V(Af) 4: ) — > lf(0, x) defined via the 
Weil representation. Similarly, the component $oo(s) at 00 is associated to the 
Gaussian for the 5 -dimensional quadratic space V'(R) over R of signature (5,0) 
under the map S(V (M) 4 ) — > ioo(0, x) ■ Thus the section <E>(s) is determined by 



(0.8) 



U = UJ\ X • • • X U) r , 



5 



and is incoherent in the sense of [19]. In particular, for h G Mp&(W) , the corre- 
sponding Eisenstein series E(h, s, $) vanishes at the center of symmetry s = . 
For any (hi, . . . , h r ) G Mpi^ x . . . x Mp r ^ we put 
(0.9) 

F dl ,..., dr (hi,...,hr,*)jr pa = E E' T (i(hi,...,h r ),U,<$>) 

T G Sym 4 (Z (p )) >0 
T represented by V(KK), 
but not by V(Q P ) 
T represents 1 over Z p 

On the right, we sum over certain Fourier coefficients of the derivative at of 
the Eisenstein series for Mp& . Our second main result is the following identity 
(Corollary 9.4). 

Theorem 0.2. We have 

F du ... , dr (hi, ...,h r , = cw 5 d ( 2 (hi) . . . w%\h r ) ■ iogp- 

(0.10) -vol(pr(K))- < Z(di,ui),... ,Z(d r ,u r ) >P roper , 

where c= ±vo\(SO(V'(R))) . 



Unexplained notation may be found in the body of the text. The identity is proved 
by unravelling both sides of (0.10), where, for the right side, we use the decom- 
position (0.4) and Theorem 0.1. The identity then reduces to the statement that, 
for T G Sym 4 (Z( p )) >0 such that T is not represented by V(Q P ) and where T 
represents 1 over 7L V , we have 



(0.11) 



(logp) 1 



Wf p (e,0,$ p ) 



vol(K)- 1 ■ I TJ {<pf) =<Z(T,u)>l 



proper 



Here, in the first factor on the left, there appears a quotient of the derivative at 
of a certain Whittaker function for the quadratic space V(Q P ) by the value at 
of a Whittaker function for a twist V(Q P ) , and, in the second factor, a Fourier 
coefficient of a theta integral. It turns out that the first factor equals the multiplicity 
e(£) of any point £ G Z(T,u>) (which is constant), while the second factor is equal 
to the number of points in Z(T,u>) . For the multiplicity e(£) , the calculation 
can be reduced to a problem on one-dimensional formal groups of height 2 which 
has been solved by Gross and Keating [7]. For the calculation of the Whittaker 
functions we use the results of Kitaoka [14] on local representation densities. It 
should be pointed out that we are using here the length of the local ring Oz,£ 
as the multiplicity of a point £ , whereas the sophisticated definition would also 
involve Tor-terms. It is a fundamental question whether these correction terms 
vanish. This question we have to leave open. 
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In summary, we may say that Theorem 0.2 is proved by explicitly computing both 
sides of (0.10) and comparing them. It would of course be highly desirable to find 
a more direct connection between the analytic side and the algebro-geometric side 
of this identity. 

We now give an overview of the structure of this paper. In section 1, we introduce 
the Shimura variety and formulate the moduli problem solved by M. . Our special 
cycles are introduced in section 2. We define the fundamental matrix in section 3 
and isolate there the part of Z lying purely in characteristic p . It is clear from 
the above description that to proceed further we need a thorough understanding of 
the supersingular locus of M x SpecZ(p) SpecF p . This is essentially due to Moret- 
Bailly [23] and Oort [24]. In section 4, we give a presentation of their results in 
terms of Dieudonne theory, better suited for our needs. A similar presentation 
was independently given by Kaiser [11] for a different purpose. The heart of the 
paper is section 5. In it we determine the space of special endomorphisms of certain 
Dieudonne modules and deduce the characterization of isolated intersection points 
(Theorems 5.11, 5.12 and 5.14). Here again the exceptional isomorphism plays 
a vital role. In section 6, we explain the reduction of the calculation of e(£) to 
the result of Gross and Keating, and, in section 7, we explain how to count the 
number of isolated points. Section 8 is a review of the Fourier coefficients of Siegel 
Eisenstein Series. In section 9, we bring everything together and prove the identity 
(0.10) above. In section 10, we review some results of Kitaoka and show how 
they can be used to prove the formulas on Whittaker functions needed in section 
9. Finally there is an appendix containing some facts on Clifford algebras in our 
special situation. 

In conclusion we wish to thank A. Genestier for very useful discussions on our 
special cycles which helped us to correct some misconceptions we had about them. 
We also thank Th. Zink for helpful remarks. We thank the NSF and the DFG for 
their support. S. K. would like to express his appreciation for the hospitality of 
the Univ. Wuppertal and the Univ. of Cologne during January 1995 and May and 
June of 1997 respectively. Finally, M. R. is very grateful to the Math Department 
of the University of Maryland for inviting him and making his stay in Washington 
a memorable pleasure. 
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§1. The Shimura variety. 

In this section, we review the construction of the Siegel 3-folds associated to 
indefinite quaternion algebras over Q , and the corresponding moduli problem. 
The use of the Clifford algebra is modeled on [28]. We refer to the appendix for 
some facts on those Clifford algebras that will be relevant for our purposes. 

Let B be an indefinite quaternion algebra over Q , let C = M 2 (B) , with 
involution x' = t x L , and let 

(1.1) V = {x eC | x' = x and tr(x) = }. 

We define a quadratic form q on V by setting x 2 = q(x) ■ I2 G M 2 (B) , cf. Appen- 
dix, A. 3. Since B is indefinite, the signature of (V, q) is (3, 2) , cf. Appendix, A. 6, 
and the Witt index of V over Q is 2 if B = M 2 (Q) and 1 if B is a division 
algebra, cf. Appendix, A. 3. Let C(V) be the Clifford algebra of the quadratic 
space (V,q) . Since, for x G V C C , x 2 = q(x) , there is a natural algebra homo- 
morphism C(V) — > C extending the inclusion of V into C . The restriction of 
this map to the even Clifford algebra C + (V) induces an isomorphism 

(1.2) C+(V)~C. 
Let 

(1.3) G = GSptn(V) = {geC x \ gg' = v{g) }, 

cf. Appendix, A. 3, so that G is a twisted form over Q of GSp^ , cf. Appendix, 
A. 2. The group G acts on V C C by conjugation and this action yields an exact 
sequence 

(1.4) 1 — > Z — > G — ► SO(V) — > 1, 
where Z is the center of G . 

Let V be the space of oriented negative 2 -planes in V(R) . This space has 
two connected components and the group G(M) acts transitively on it, via its 
action on V(R) . For an oriented 2 -plane z G V , let z\ , z 2 G z be a properly 
oriented basis such that the restriction of the quadratic form q from V(R) to z 
has matrix —1 2 for the basis z\ , z 2 . Let j z = z\z 2 G C(R) . Viewing j z as the 
image of the element ziz 2 G C(V(R)) , the Clifford algebra of V(R) , and recalling 
the commutative diagram of section A. 3 of the Appendix, we see that j' z = —j z 
and that j 2 = —z\z\ = —1 . Hence, j z j z = 1 and so, j z G G(M) . There is an 
isomorphism of algebras over R , 



(1.5) 



C C + (z) i I— > ziz 2 , 
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where C + (z) is the even Clifford algebra of the real 2 -plane z . The composition 
of this map with the map 

(1.6) C+(z) c C + (V(R)) C(R) = M 2 (B(R)) 

induces a morphism, defined over R, h z : S — > G, where § = i?c/R^m , as 
usual. Note that h z {i) = j z . The space V can thus be viewed as the space of 
conjugacy classes of such maps under the action of the group G(R) . The data 
(G,T>) or (G,h z ) defines a Shimura variety Sh(G,T>) , [2], [3], whose canonical 
model is defined over Q . Note that T> is isomorphic to two copies of the Siegel 
space of genus 2 , and, if B = M 2 (Q) , Sh(G, V) is just the Siegel modular variety 
of genus 2 . 

Since G satisfies the Hasse principle, the Shimura variety represents a certain 
moduli problem over (Sch/Q) , [17]. To define this we must introduce more nota- 
tion. 

Fix a maximal order 0^ in B such that O l b = Ob , and let Oc = M 2 (Ob) ■ 
Let D(B) be the product of the primes p at which B p is division, and, as in 
[1], choose t e B x such that r L = -r , r 2 = -D(B) , and tObt' 1 = O b ■ By 
section A. 7 of the Appendix, the map x h-> x* = tx l t~ x is a positive involution of 
B preserving Ob ■ Also, for 

(1.7) a = ( T r ) e M 2 (B), 

a' = —a and x* = ax' aT 1 = a~ x x' a is a positive involution of C , preserving 
Oc- 

Let U = Oc , viewed as a module for Oc under both left and right multiplica- 
tion. Define an alternating form: 

(1.8) < , > : U x U — > Z 
by 

(1.9) < x, y > = triy'a^x). 
Then 

(1.10) < cx, y > = tr(y'a~ 1 cx) = tr(y'a _1 co;Q; _1 a;) = < x, c*y >, 
and 

(1.11) < xc, y > = tr(y' a~ x xc) = tr(q/ a -1 x) = < x,yc > . 
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Thus, if g e G , 

(1.12) < xg,yg >= v{g) < x,y >, 
and, in particular, for z e T> , 

(1.13) < xj z ,yj z > = < x,y > . 

We fix a compact open subgroup K C G(Af) . The functor Mr- associates to 
5 G (Sch/Q) the set of quadruples, (A, t, A, 77) , up to isomorphism, where 

(i) A is an abelian scheme over S , up to isogeny, 

(ii) i : C — > End (A) is a homomorphism such that 

det(t(c);Lie(A)) = iV°(c) 2 , 

where N°(c) is the reduced norm on C . 
(hi) A is a Q -class of polarizations on A which induce the involution * on C : 

A o 1(c) o A -1 = t(c*). 

(iv) fj is a K -class of isomorphisms 

77 : 1/(A) U<8)Af 

which are C -linear (for the left module structure on U ) and respect the 
symplectic forms on both sides up to a constant in . Here 

V(A) = Y[T e (A)®Q. 

t 

For the precise meaning of the datum (iv) we refer to [17], p. 390. In particular, 
if S = Spec k is the spectrum of a field, the K -class fj is supposed to be stable 
under the action of the Galois group Gal(fc/fc) where k is the algebraic closure 
used to form the Tate module of A . 

Note that the abelian scheme A will have relative dimension 8 over S . 

Proposition 1.1. For K neat this moduli problem is representable by a smooth 
quasi-projective scheme Mk over Q and 

Mjf(C) ~ Sh(G,V)(C). 

Proof. For the representability, see [17]. We prove the last assertion in detail, since 
the conventions involved will be used later. 
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For t G B x , as above, let 

(1.14) t = D(B)~h e B X (R), 
so that tq = —1 . Choose G B x such that 

(1.15) 0r = -t0, and l = -0. 
Since B is indefinite, (3 2 > 0, and we can set 

(1.16) o = (0 2 )^0eB x (R), 
so that 0q = 1 . The vectors 

(_ A *). »d (_ To/3o *"*) 6 V<*) 
form a standard basis of an oriented negative 2 -plane zq G £> , and 

^> = *)U«. To/3(l ) = ( T ° T y°i*r 

Lemma 1.2. For any z 6 D, 

< > = < >, 

and, /or x G U(R) , x^O, 

< ay^a; > > 0, 

if z lies in the same connected component of V as zq , and 

< xj z ,x > < 0, 
if z and zo lie in opposite components. 

Proof. For the first assertion: 

(1.19) < xj z , y > = - < x, yj z > = < yj z , x > . 

For the second, write z = gz for g G G(M) , so that 
(1-20) j z = gj Za g~ x = gaog- 1 . 

Then, we have 

< xj z , x > = < xga Q g~ x , x > 

= KfiO" 1 < xga ,xg > 
(1.21) = v(g)~ 1 tT((xg) , a~ 1 xga ) 

= v(g)~ 1 D(B)~^ triaixgYa-'ixg)) 
= u(g)- 1 D(B)-hv((xgr(xg)). 
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Since ihi* is a positive involution, this gives the claim. □ 

Let T> + be the connected component of T> containing zq and T>~ the con- 
nected component of V not containing zo . Then, for any z G , we obtain a 
(principally) polarized abelian variety over C , 

(1.22) A z = (U(R),j z ,U(Z),±< , >) 
with dimA 2 = 8 and with an action 

(1.23) i : O c ^ End(A z ). 
Note that 

(1.24) V(A g ) = U(Z)®Q = U(A f ). 
If 

(1.25) 7 G T = { g G G(Q)+ | U(Z)g = U(Z) }, 
then right multiplication by 7 _1 induces an isomorphism 

(1.26) A z A 1Z . 

Thus T\V + parametrizes such principally polarized abelian varieties, up to iso- 
morphism. 

More generally, to (z,g) G V x G(Af) , we associate the collection (^4,i,A, fj) 
defined by: 

• (A, l) = (A z , l) , where A z is taken up to isogeny. 

• A is the Q -class of polarizations determined by < , > . 

• fj is the K -class containing the isomorphism: 

r(9) 

V{A Z ) = U(A f ) U(Af). 

Note that, if 7 G G(Q) and k G K , then (72, ^gk) defines a collection isomorphic 
to that defined by (z,g) , via the element of Hom°(A z , A JZ ) given on U(M) by 
right multiplication by 7 _1 . The map 

(1.27) G(Q)(z,g)K»(A,L,\,fj)/~ 
yields the isomorphism 

(1.28) G(Q)\V x G(A f )/K M K (C). □ 
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We now turn to the construction of a p -integral model. Fix a prime p such 
that p \ D(B) , so that C <E> Q p — M^Qp) . Let Oc be the maximal order chosen 
above, and note that the maximal order Oq CS> Z p in C <E> Q p is the stabilizer of 
the lattice U% = U <E> Z p in t/ <g> Q p under both right and left multiplication. The 
choice of r made before (1.7) ensures that < , > defines a perfect pairing 

(1.29) < , >: U Zp x U Zp — > Z p . 

Let -Kp be the stabilizer of t/z in G(Q P ) , acting on L/q via right multiplication. 
Let K p C G(A P f) be compact open, and take K = K p ■ K p . 

We now want to formulate a moduli problem over (Sch/Z( p )) which extends 
the previous one. The functor Mkp associates to S e (Sch/Z( p )) the set of 
isomorphism classes of quadruples (A, t, A, f) p ) where 

(i) A is an abelian scheme over S , up to prime to p isogeny 

(ii) i : Oc®^{ p ) — > End(A)®Z(p) is a homomorphism such that, for c G Oc , 

det(t(c);Lie(A)) = A^°(c) 2 , 

where is the reduced norm on C . 

(iii) A is a Z^-class of isomorphisms A — > A such that nX , for a suitable 
natural number n , is induced by an ample line bundle on A . 

(iv) f) p is a K p -class of Oc* -linear isomorphisms (in the sense of Kottwitz) 

rf : V p (A) -^U®A p f , 

which respects the symplectic form on both sides up to a constant in (Ay) x . 
Here 

V p (A) = l[T e (A)®Q. 

In the determinant condition above, the equality is meant as an identity of 
polynomial functions. In the case at hand, it simply says dim A = 8 . 

Proposition 1.3. For K p neat the above moduli problem is representable by a 
smooth quasiprojective scheme Mkp over Spec Z( p ) . Its generic fibre can be 
canonically identified with Mk , 

M K p x Spec Z(j>) Spec Q = M K . 

Let us briefly explain the last identification on geometric points. Let S be 
the spectrum of an algebraically closed field of characteristic . Let us consider 
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(A, l, X,f] p ) G M.kp{S) . Then the p-adic Tate module T P (A) is equipped with 
a perfect symplectic form, unique up to scaling by Z* and hence there is an 
Oc <E> Zp-linear isomorphism 

V P ■ T P (A) U Zp , 

which respects the symplectic forms up to Z* . The set of such 1] P 's form a single 
orbit for K p , which acts via right multiplication in U% . Hence, from (A, t, A, fj p ) , 
we obtain an object (A <g> Q, t <8> Q, A <8> Q, ff ■ fj p ) of Mk(S) . Passage in the other 
direction is similar. For example, in the isogeny class A and for rj e fj , there is an 
abelian variety B , unique up to prime to p isogeny, such that rj p (T p (B)) = U% p . 

The above proposition tells us that, when K = K p ■ K p , as above, then Mrp 
provides us with a smooth model of Sh(G, D)k over Z( p ) . From now on, we will 
use the same notation for both moduli problems, if this does not cause confusion. 
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§2. Special cycles. 

In this section we give a modular definition of the special cycles in Sh(G,T>) , 
which were defined analytically in [18]. We then explain the relation between the 
two definitions. 

Recall that the quadratic form on the space V C C = M 2 (B) was defined by 
x 2 = q(x) ■ I2 • Let 

(2.1) (x,y) =q(x + y) -q(x) -q(y) 

be the corresponding bilinear form, so that q(x) = | (x, x) . If x = (x\, x 2 , • • • , x n ) G 
V n (Q) , we let 

(2-2) q(x) = ^((x it x j )) i , j eSym n (Q). 

This defines a quadratic map q : V n — > Sym n . 

Fix a positive integer n . For d G Sym n (Q) a symmetric rational matrix, let 

(2.3) fi d = { x G V n I q(x) = d } 

be the corresponding hyperboloid. The group G acts diagonally on V n and pre- 
serves fid . 

Cycles in Sh{G, V) were defined analytically in [18] as follows. For x G fid (Q) > 
let < x > C V be the Q -subspace spanned by the components of x , and let 
V x = < x >- L be its orthogonal complement. Let V x denote the space of oriented 
negative 2 -planes in \4(R) , and let G x be the pointwise stabilizer of < x > in 
G . Note that G x ~ GSpin(V x ) , and that V x C V . Moreover, for z G V x , the 
homomorphism h z factors through G X (M.) . Thus there is a natural morphism of 
Shimura varieties, rational over Q , 

(2.4) Sh(G x ,V x ) — > Sh(G,V). 

If the space < x > is not positive-definite, then V x = . If < x > is positive- 
definite of dimension r then d is positive semi-definite of rank r , sig(V x ) = 
(3 — r, 2) and T> x has codimension r in T> . Hence the previous construction is 
only interesting when d is positive semi-definite and even only when d is positive 
definite with n < 3 . 

For a fixed compact open subgroup K C G(Af) and for ft G G(Af) , there is a 
cycle, namely the image of the map 
(2.5) 

Z(x, h- K) : G X (Q)\V X x G x (A f )/(G x (A f ) n /iK/i" 1 ) — > G(Q)\D x G(A f )/K 
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given by (z, g) \— > (z, gh) . This map is finite and generically injective, hence the 
cycle image is taken with multiplicity 1. This cycle of codimension r = rk(d) on 
Sh(G, V) K is rational over Q . 

Assume that fi d (Q) ^ (f> and fix x E O d (Q) . Let <p E S(V(A f ) n ) K be a 
Schwartz function which is K -invariant, and write 

(2.6) su PP (v?) n n d (A f ) = ]jKh; 1 x 

r 

for elements h r E G(Af) . Then define the weighted cycle: 

(2.7) Z(d, ^K) = J2 Vih^xo) ■ Z(x , h r ; K). 

r 

This cycle is independent of the choice of xq and of the orbit representatives h r . 
It is a (weighted linear combination of) cycle(s) of codimension r = rk(d) on 
Sh(G, V)k and is rational over Q . 

If ip is the characteristic function of a K -invariant compact open subset uj of 
V(Af) n , then Z(d, u>; K) = Z(d,(p;K) can be considered as a disjoint union of 
maps (2.5), or as the union of the images of these maps. 

We introduce the following definition, which will play a key role throughout the 
paper. 

Definition 2.1. Let (A, i, \,rj) E M K (S) . A special endomorphism of (A, t, \ : rf) 
is an element j E End^A, l) which satisfies 

(2.8) j* =j and tr°(j) = . 

Here * denotes the Rosati involution of A . Also note that End (A, l) is a 
finite-dimensional semisimple Q -algebra, so that the reduced trace appearing here 
makes sense. Indeed, this is well-known when S is the spectrum of a field. The 
case when S is irreducible follows by reduction to its generic point, and the general 
case follows by considering the irreducible components of S . 

Lemma 2.2. Let j be a special endomorphism of (A, t, A,?j) E Mk(S) , where S 
is connected. Then 

(2-9) f = q(j)- id , 

with q(j) E Q . 

Proof. Again we may reduce first to the case where 5 is irreducible and then to 
the case when S is the spectrum of a field. However for n E fj let x = rj*(j) E 
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Endc{U(Af)) = C(Af) . Under the last identification the adjoint involution * 
w.r.t. < , > corresponds to the involution ' on C(A) , cf. (1.11). Hence x lies in 
V(Af) and the assertion follows, cf. appendix A. 3. □ 

The previous Lemma justifies the following definitions. Let S be a connected 
scheme and £ = (A, t, X,fj) G Mk(S) be an S -valued point of Mk ■ Let 

(2.10) Cl = End° s (A,L) op 
and 

(2.11) if = {x G Cj | x* = x and tr°{x) = 0} . 

Then is the finite-dimensional Q -vector space of special endomorphisms with 
quadratic form 

(2.12) g € : — > Q 

given by x 2 = ^(x) • id a . By the universal property of the Clifford algebra of 
(V®, q^) there is a natural homomorphism 

(2.13) C(V°,qs) — Cj • 

This structure is compatible with specialization. If S' C S is a connected closed 
subscheme, let £' G Mk(S') be the restriction of £ . Then we have a homomor- 
phism of Q -algebras 

(2.14) Cf = End£(A, 0° p End S '(A t) op = C%, 
inducing a map 

V° V° 

of quadratic spaces. 

Let us spell out these concepts in the classical case. 

Lemma 2.3. Let £ G M K (C) with parameter (z,g) in Sh(G,V) K . Let Al° p = 
U(M.)/U(Z) be the real torus underlying A z . 

(i) 

C(Q) End°(4°V)° P , y->r(y), 

where r(y) denotes the action of y G C(Q) on U(M) D U(Q) by right multiplica- 
tion. Moreover, r(y)* = r(y') . 
(ii) 

Cf ~ Cent cm (j z ), and V° ~ { x G V(Q) \ xj z = j z x }. 
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(Hi) 

Cent c( m ) (j z )nV(R) = z ± . 

In particular, 

vl = v(Q)nz ± , 

and so < dimQ < 3 . 

Proof. The first two assertions are obvious by (1.11). To prove the last assertion let 
z\ , Z2 E z be a properly oriented basis such that the restriction of the quadratic form 
q to z has matrix — 1 2 in terms of this basis. Let v E V(R) with (f,^) = ai , 
i = 1, 2 . Then 

v • j« = v • (zi • z 2 ) = z\z 2 v - a 2 z\ + a\z 2 = j z v - a 2 z\ + a\z 2 . 
Hence v E Centc(R){jz) iff a>i = a 2 = , i.e. iff v E z . □ 

Let us return to the abstract situation. 

Lemma 2.4. Let £ = (A, t, X,fj) E Mk(S) be a point with values in a connected 
scheme S . The quadratic space V? is positive- definite. 

Proof. We may assume that S is the spectrum of a field. The assertion follows 
from the positivity of the Rosati involution, since 

q(x) ■ io\a = x 2 = x ■ x* , x E V® . □ 

We next give a modular definition of the cycles introduced above. We take 
here the point of view that a cycle is given by a finite unramified morphism into 
the ambient scheme. Let K C G(Af) be a compact open subgroup, and let 
<jj C V(Af) n be a K -invariant compact open subset. Consider the functor on 
(Sch/Q) which associates to a scheme S the set of isomorphism classes of 5- 
tuples (A, l, A, 77; j) where (A, t, A, 77) E Mk(S) . Here the additional element j = 
• • • 7 in) E End°(A, i) n is an n-tuple of special endomorphisms of A , satisfying 
the following conditions. 

(2.15) For some (and hence for all) rj E fj , the element 77* (j) E Endc(U(Af)) n 
lies in u> . 

(2.16) q(j) = d. 

Let us explain the condition (2.15). As in the proof of Lemma 2.2 above, for any 
i]Er] 

x = r,*(j) e V(A f ) n C C(Af) n = End c (U(A f )r • 
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The condition imposes that x G ui . If rj is changed to r(k) o rj , with k G K and 
r(/c) G Endc(U (Af)) the endomorphism defined by right multiplication by k , then 

(2.17) (r(k) o n)*(j) = r(k) o 77* (j) o r(k)-\ 

The condition (2.15) asserts that = r(x) for some x G to . If this is the case, 

then 

(2.18) (r(k) o v )*(j) = r(k) o 77* (j) o r(A;) _1 = r(fc _1 a;fc), 

and k~ x xk G cj . Thus the condition (2.15) depends only on fj . 
To interpret condition (2.16) we may assume S to be connected. Let ( , ) be 
the bilinear form on the space of special endomorphisms of (A, l, A, rj) associated 
to the quadratic form q of lemma 2.2. Then q(j) = G Sym n (Q) is 

defined as in (2.2). The condition (2.16) requires that q(j) = d. 

Proposition 2.5. The above functor has a coarse moduli scheme Z(d, u>) . If K 
is neat, then Z(d,u) is a fine moduli scheme and the forgetful morphism 

(2.19) Z(d,u) — > M K 

is finite and unramified. Furthermore Z{d, uj)(C) = Z(d, u, K) . 

Proof. The first statement follows easily from the second. Let us assume that K 
is neat. The relative representability of the forgetful morphism by a morphism of 
finite type follows in a standard way from Grothendieck's theory of Hilbert schemes 
since Mk may be considered as a moduli scheme of polarized abelian varieties with 
additional structure. By the Neron universal property the valuative criterion for 
properness is satisfied. Since the matrix d gives the squares jf of the special 
endomorphisms, the morphism is quasi-finite and hence finite. The unramifiedness 
follows from the rigidity theorem for abelian varieties. 

The last statement is to be interpreted as an equality between the image of (2.5) 
and Z(d,uj)(C) , and follows easily from Lemma 2.3 above. 

We now assume that p \ 2D(B) and that K = K p ■ K p with K p neat, as 
in Proposition 1.3, and we formulate a p -integral version of the previous moduli 
problem. 

Before doing this let us point out that for a point £ = (A, t, \ ff) G M.kp{S) 
of the p -integral version of our moduli problem with values in a connected scheme 
S we may transpose the concepts above. Hence we introduce the Z( p ) -algebra 



(2.20) 



C c = End s (A, 0° P ®^( P ) 
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and 

(2.21) V£ = {x G C € ; = x and tr°(x) = 0} . 

The latter is a Z( p ) -module with a Z( p ) -valued positive definite quadratic form. 
The elements of will again be called the special endomorphisms of (A, t, A, rf) . 

Let now again d G Sym n (Q) . Let cu p C V(A P j.) n be a if p -invariant open 
compact subset. Then a point of the corresponding moduli problem Z(d, u p ) on a 
Z(p) -scheme £ is an isomorphism class of 5-tuples (A, t, A, j) where (A, t, A, ?? p ) 
is an object of «M.k-p(<S) and where j G (End(A, i)(g>Z( p )) n is an n -tuple of special 
endomorphisms which satisfies (2.16) above and, in addition, 

(2.22) (rf)*(j) 6w". 

These conditions are to be interpreted in the same way as (2.15)-(2.16) above. 

To clarify the relation between the p -integral version Z(d, u p ) and the previous 
Z{d, u) , let 

(2.23) u p = V(Z p ) n 

where V(Z P ) = V{Q_ p )n{Oc®Z p ) , the intersection taking place inside of C®Q P . 
Let 

(2.24) uj = u p xuj p , 
a K -invariant open compact subset of V(Af) n . 

Proposition 2.6. If K v is neat, the functor Z(d, u p ) is representable by a scheme 
which maps by a finite unramified morphism to M.kp ■ Furthermore, there is an 
identification 

Z(d, u p ) x spec z (j0 Spec Q = Z(d,u) . 

Remark 2.7: By Lemma 2.4 the scheme Z(d, uj p ) is empty unless d is positive 
semi-definite. Similarly Z(d, u p ) = 0, unless d G Sym n (Z( p )) . Note that it 
may well happen that Z(d,u p ) is non-empty but where both sides of the equality 
in Proposition 2.6 are empty. In fact we will later consider cases in which d G 
Sym 4 (Z( p )) is positive definite so that Z(d,ou) = and when Z(d, uj p ) ^ . 

^From now on, since we will be interested in the arithmetic situation, we will 
simplify our notation by denoting u what is denoted by u p above, i.e., 

(2.25) to C V(A p ) n 
is a K p -invariant open compact subset. 
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§3. The intersection problem. 

We continue to fix p\ 2D(B) and a neat open compact subgroup K p C G(A^) 
as at the end of section 2. Then M = Mkp is a regular noetherian scheme of 
dimension 4. We wish to consider the intersection of the cycles introduced in a 
modular way in the previous section. Let us set up our problem in a more precise 
way. 

We fix integers ni, . . . , n r with 1 < rii < 4 and with ni + - ■ •+n r = 4 . For each 
% , we choose d{ G Sym ni (Q) positive definite, and a K p -invariant open compact 
subset u t cy(Aj) n -. Let 

(3.1) Z = Z(d 1 ,u> 1 ) x M ■ ■ ■ x M 2(d r ,oj r ) 
be the fiber product of the corresponding special cycles. 

By what has been said in section 2, since the codimensions of the generic fibres 
of our special cycles add up to the arithmetic dimension of .M , one might expect 
that Z consists of finitely many points of characteristic p . We will see that this 
is in fact quite false, but we will be able to determine that part of Z which lies 
purely in characteristic p and also determine the isolated points of Z . 

Let £ be a point of Z , with corresponding point (A^ t, \ rf ) E M . We denote 
by and (Vg, q^) the Z( p ) -algebra and the quadratic Z( p ) -module associated to 
(A{, i, A,rf) , cf. (2.20). The projections Z — > Z[d^ Ui) define -tuples of special 
endomorphisms 

(3.2) jiGV^ , i=l,... ,r . 
Let 



(3-3) T C = 1 



/(jljl)? ••• (jlJr)A 



\(jrjl)€ ••• (jrJrOe/ 



G Sym 4 (Z {p) ), 



where ( , )^ is the bilinear form associated to . Here, as always, p ^ 2. The 
matrix is called the fundamental matrix associated to the intersection 

point £ of the special cycles Z(d\, ui\), . . . , Z(d r , u r ) . We note that the blocks on 
the diagonal of are d±, . . . , d r . By the results of section 2, the function £ i— > 
is constant on each connected component of Z . Therefore, for T G Sym 4 (Z( p )) 
we may introduce 

Zt = (Z(di, ui\) fl • • • fl Z(d r , iv r ))T = 
(3.4) union of the connected components of Z 

consisting of the points £ with T^ = T 
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We note here the hereditary nature of our construction, given by 

(3.5) Z(T, wi x • • • x u r ) = (2(di, ui\) xj^ ■ ■ ■ xm Z(d r , uJ r ))T , 

valid provided that the blocks on the diagonal of T are d±, . . . ,d r . We may 
therefore write 

Z = Z(di, u)\) x M . . . x M Z(d r , u) r ) 

T 

(3.6) = ]J Z{T,u) . 

T G Sym 4 (Z (p) )> 
diag(T) = (d 1 ,...,d r ) 

Here u = uii x . . . x u r . 

We shall see that the fundamental matrix governs the intersection behaviour of 
our special cycles. We first note the following result. 

Proposition 3.1. Let £ G Z = Z{d\,u)\) x M ... x M Z(d r ,uj r ) where Ui C 
V(A P f) nt and di G Sym n . (Q) positive- definite with ri\ + . . . + n r = 4 . Suppose 
that det(T^) ^ . Then £ lies in the special fiber of Z , and £ does not lie in the 
closure of any point of Z in the generic fiber. 

Proof. By Lemma 2.4 the assumption on T ? means that T ? G Sym 4 (Z( p )) is 
positive definite. However, for a point of M in characteristic zero, the space of 
special endomorphisms is contained in a 3-dimensional positive definite quadratic 
space. □ 

Next suppose that £ G M(¥ p ) . In this case, the standard Honda- Tate results 
yield information about the possibilities for C° . We have i : M 2 (B) = C 
End°(A^) , so that, up to isogeny ~ A x A where dim A = 4 and there is an 
embedding B ^ End (A) . 

Lemma 3.2. Suppose that p \ D(B) . Then there are no simple abelian varieties 
Aq over F p with dimAo = 2 or 4 and with B <—> End°(Ao) . 

Proof. If A is simple over ¥ q , then E = End°(A ) is a central simple algebra 
over F = Q(tva ) , and 



(3.7) 



2dimA = [E : F}% ■ [F : Q]. 
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Here tta denotes as usual the Frobenius endomorphism. If dimAo > 2 and Aq 
remains simple over ¥ p , then F is a CM field. Suppose that dimAo = 2 , so that 
[F : Q] = 2 or 4 . The second case is excluded, since then E = F is commutative. 
In the first case, E is a division quaternion algebra over F ramified only at places 
over p. Thus p splits in F and inv v (E) = inVy(E) = ^ for v \ p. But the 
embedding B ^ E yields an isomorphism B ®q F ~ F . This is possible only if 
p | -D(-B) and F splits B at all other primes. 

If dim A = 4 , then [F : Q] = 2 , 4 or 8 , and the last case is again excluded 
since E = F . In the case [F : Q] = 4 , E is a quaternion algebra over F , ramified 
only at primes lying over p , and B ®q F ~ F . This cannot occur if p { D(B) . 
Finally, if [F : Q] = 2 , then p splits in F and F is a division algebra over F 
of dimension 16 with invariants \ and | at the primes over p . There is no 
homomorphism from a quaternion algebra B ®q F into such an algebra. □ 

Returning to A , and assuming that p \ D(B) , we see that A cannot be simple 
and that any simple factor of A of dimension 1 or 2 must occur with multiplicity 
at least 2 . Thus we have various possibilities for A , up to isogeny: 

(3.8.i) A ~ A 2 x A 2 , with dimA 2 = 2 simple and End°(A 2 ) ~ F for a CM field 
F with [F : Q] = 4 which splits B , i.e., such that B <g> Q F ~ M 2 (F) . 
Then, End (A) ~ M 2 (F) , Cf = End (A, t) ~ F , and = Q . 

(3.8.ii) A ~ A 2 x A 2 , with dimA 2 = 2 simple and End°(A 2 ) ~ F , where F is a 
quaternion algebra over a CM field F with [F : Q] = 2 . More precisely, p 
splits in F and E ~ H p ®q F , where H p is the quaternion algebra over 
Q ramified at oo and p . Let B' be the quaternion algebra over Q whose 
invariants agree with those of B except at oo and p. Then End (A) ~ 
M 2 (F) , End°(A, l) ~ B' ®qF , and V? = {x G B' \ tr(x) = 0} . Here note 
that FOqF' ~ M 2 (H P ) and hence that (FOqF) O f (B'®qF) ~ M 2 (F) . 

(3.8.iii) A ~ Aq x where A and are non-isogenous ordinary elliptic curves. 
Then End°(A) ~ M 2 (F ) x M 2 (Fi) for imaginary quadratic fields F and 
Fi , which split B . Then, End (A, l) ~ F x Fi , and = Q . 

(3.8.iv) A ~ Aq , for an ordinary elliptic curve Ao . Then End°(A) ~ M^Fq) 
where the imaginary quadratic field Fq splits F, End (A, l) ~ M 2 (Fo) 
and ~{i;6 M 2 (F ) \*x = x, tr(x) = 0} . 

(3.8.v) 4 ~ Ag x , where Ao is a supersingular elliptic curve and A\ is an 
ordinary elliptic curve. Then End (A) ~ M 2 (H P ) x M 2 (Fi) , End (A, t) ~ 
B' xFi . Since the Rosati involution acts on End (A, i) by (6, a) i— > (6^, a) , 
the conditions x* = x and tr(x) = force ~ Q . 

(3.8.vi) A ~ Aq , for a supersingular elliptic curve A . Then End (A) ~ M 4 (F P ) , 
End°(A,i) ~ M 2 (F') , and 

= {xe M 2 (B') | = V = x, tr(x) = 0} = V. 
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For the last identification we are using the proposition in section A. 4 of the 
Appendix. Indeed, the Rosati involution on End°(A^) ~ M%(H p ) is of main type 
by A. 4 and its restriction to M2(B) ofnebentype. Hence the involution on M2 (-£?') 
is indeed of main type. 

Note that dim V? < 3 , with the exception of the supersingular case (3.8.vi). As 
a consequence, we have the following: 

Proposition 3.3. Let T G Sym 4 (Z^) and uj C F(A^) 4 with corresponding 
special cycle Z(T,u) . If det(T) 7^ 0, then the point set underlying Z(T,u>) maps 
to the supersingular locus of M. xs pe cZ {p) SpecF p . In particular, Z(T,u>) is proper 
over SpecZ( p ) with support in the special fibre. 

Proof. Indeed the previous results imply that this is true for closed points. 

Corollary 3.4. For i = 1,... ,r, let di G Sym n . (Q) be positive definite with 
ni+. . .+n r = 4 , and let uji C V(A^) ni with corresponding special cycles Z(di, u>i) . 
For T G Sym 4 (Z( p )) with diagonal blocks d\, . . . , d r , let Zt be the union of the 
connected components of Z(d\,oj\) x M . .. x M Z{d r ,u r ) where the fundamental 
matrix has value T . If det(T) 7^ then the point set underlying Zt lies over the 
supersingular locus of M. xs pecZ(p) SpecF p . □ 

Having answered these very crude questions on the intersection behaviour of our 
special cycles, we are led to ask more precise questions. Again for i = 1, . . . , r let 
di G Sym n . (Q) be positive-definite with n\ + . . . + n r = 4 and let Ui C V{K p ^) ni 
with corresponding cycles Z{d\,u\), . . . , Z{d r , uj r ) . We then ask: 

a) Under which conditions do the cycles Z(d\, cui), . . . , Z(d r , u r ) intersect prop- 
erly? More precisely, can one parametrize the isolated points of Z = Z(d\, 00 1) x_a/[ 
. . . Xj^i Z(d r , u r ) and calculate at such an isolated point y , 

(3-9) e(y) = \g 0z JO z , y ) ? 

b) Let Y be a connected component of Z = Z(di,u>i) x^i ••• x m Z(d r ,u r ) 
lying over the supersingular locus of M. x SpecZw SpecF p . The intersection number 
along Y is 

(3-10) x{Y,0 Zi ^ 0m ...^ 0m O Zt ) , 

cf. [22], [27]. An important question to answer is when the derived tensor product 
here can be replaced by an ordinary tensor product, i.e. by O z ■ In the case when 
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Y is an isolated point this would mean that the length in (3.9) is in fact the 
intersection number of Z\,... ,Z r at y . In particular one may ask, when does 
the intersection number along Y depend only on the scheme Y ? Related to this 
question is the problem of the singularities of the schemes Z(d,co) : under which 
conditions are they Cohen-Macaulay, or even locally complete intersections? In 
general they are neither [21]. 

Our next task will be to investigate the structure of the supersingular locus 

M ss CM x Spec Z(p) Spec F p . 
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§4. Structure of the supersingular locus. 

As mentioned in the introduction, the results of this section are a presentation of 
results of Moret-Bailly [23] and Oort [24] . A similar presentation was independently 
given by Kaiser [11]. 

We put F = ¥ p , and let W = W(¥) be the ring of Witt vectors of F and 
K = W <g>z p Q P its quotient field. Also write W[F, V] for the Cartier ring of F . 

Throughout this section, we assume that p \ D(B) , and we fix an isomorphism 

O c ®z % P ^ M 4 (Z p ) . 

Suppose that £ = (A, l, \,fp) G A^1 SS (F) , and let A{p) be the p-divisible (for- 
mal) group of A. The action of Oc ®z Z p — M 4 (Z p ) on A{p) then induces a 
decomposition A(p) ~ A (p) 4 , where A (p) is a p-divisible formal group of di- 
mension 2 and height 4 . Let Lq be the (contravariant) Dieudonne module of 
Aq{p) and let C = L ®w K, be the associated isocrystal. This does not depend 
on the choice of £ G A4 SS (F) , up to isomorphism. 

More precisely, we fix a base point £ D = (A a , l q , A g , 77^) G A^ SS (F) and let C = 
Lq <S>w K< be the isocrystal associated to it. The isocrystal C has a polarization 
< , > , is isoclinic with slope \ , and has dim^; C = 4 . Then F is a -linear, 
V = pF~ x is a -1 -linear, and 

(4.1) < Fx, y > = < x, Vy > a . 

If £ = (A, l, A, rf ) G A-^ SS (F) is another point, then the choice of an isogeny between 
£ and £ defines a W -lattice L C C . 

For a W -lattice L C £ of rank 4 , set 

(4.2) L ± = {xg£ |<x,L>cW}. 

Definition 4.1. a) A W -lattice L in C is special z/ anrf only if L = c ■ L 1 - , 
for some c G JC X . Otherwise, L is non-special. 
b) A W -lattice L in C inadmissible if 

L D FL D pL . 

We note that, if L is an admissible lattice, then, since £ is isoclinic of slope 1/2 , 
we have 

dim F L/FL = 2 . 
We define a set of lattices as follows: 



(4.3) 



X = {L C £; L admissible and special} 
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If L G X then FL G X . This follows from (FL) 1 - = V~ l ■ L 1 - , cf. (4.1). 

The conditions in our moduli problem imply that the lattice L C C associated 
to £ G M ss (¥) and an isogeny between £ and £ G actually lies in X . Note that 
each admissible lattice is the Dieudonne module of a p -divisible formal group of 
dimension 2 and height 4 over F . 

Let L C C be a lattice. Then L is special or non-special depending on whether 
the generalized index [L 1 - : L~\ G Z is divisible by 4 or not. In any case this index 
is always even. If L is special and L' C L is an inclusion of index 1, then L' is 
non-special and conversely. 

If L is special, we can replace L by a ■ L , for a G /C x to obtain a lattice with 
L = L 1 - or L = pL 1 - . If L is non-special, we can scale to obtain a lattice with 
either 



with all indices equal to 2. We will call lattices scaled in this way standard. 

Recall from (3.8.vi) that End°(A ?o , t) op =: C ~ M 2 (B') , where B' is the 
definite quaternion algebra over Q with the same local invariants as B at all 
primes I ^ p . As before, let V = { x G M2(B') \ x' = x and tr(x) = } . Let 



Note that the action of G'(Q P ) on A^ o (p) up to isogeny passes to C . In fact, 
(4.6) G'(Q P ) — { <7 G GL(£) | < ^x, ^ > = !/(</) < x, y >, = }. 
Here G /C x . 

The action of G'(Q P ) preserves the set of lattices X . Fix an isomorphism B(A^) ~ 
B'(A^) and, hence, an isomorphism G(A^) ~ G'(A^) . Then, the usual analysis 
identifies G"(Q) with the group of self-isogenies of £ D and yields an isomorphism 



(4.4) 



LcL- L Cp" 1 L, or ^CiCp" 1 ! 1 , 



(4.5) 



G' = { <7GC"> X |^ 



-l 



V and ^' = }. 



(4.7) 




We will now describe the lattices in X in more detail. 



Definition 4.2. For LgI, Zei 



a(L) = dim F L/ (FL + VL) 
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Since a(L) — dimFHom W /[ F y](L,F) , we see that a(L) is the a -number, [24], 
of the p -divisible group A (p) associated to L, i.e. 

a(L) = Hom F (a p , A (p)) . 

Since 

L 

T 

FL + VL 

/ 

(4.8) FL 

\ 

FLHVL 

T 

we have 

, , f 2 if FL = VL, 
4.9 a L = <^ 

V 7 \ 1 if [L : FL + VL] = 1. 

Let 

(4.10) X = { L G X | a(L) = 2 }. 

Such lattices will be called superspecial. 

In addition to the superspecial lattices, the following type of lattice will play a 
key role in the description of the structure of X . 

Definition 4.3. A lattice L C C is distinguished if L is admissible and FL = 
cL 1 - for some c G /C x . 

We denote by X the set of distinguished lattices. Obviously, if L G X is 
distinguished, the index [L 1 - : L] is congruent to 2 mod 4. Thus L is non-special. 
Note that if L is distinguished, then FL = VL . Indeed, by (4.1) for any lattice 
L we have (FL) 1 - = V~ l L^ . Hence if FL = c-L ± we get 

L = c(FL) 1 - = cV^L 1 - = cV~ x c~ x FL = V~ X FL , 

i.e. VL = FL , as claimed. Similarly one sees that if L G X , then FL G X . 

We note that if, in the identity defining a distinguished lattice L , the order of 
c is odd, then L may be scaled to be standard in the sense of the first alternative 
of (4.4) above. If the order of c is even, then L can be scaled to be standard in 
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the sense of the second alternative of (4.4), and hence, FL can be scaled to be 
standard in the sense of the first alternative of (4.4). 

For any L G X and for any F-line £ C L/FL, let L = L(£) be the inverse 
image of £ in L . Thus 

L D L D FL 

(4-11) | | | 

L/FL D £ D 

Lemma 4.4. For £ C L/FL , L = L{£) e X . 

Proof. First, since FL = VL , we have 

(4.12) FL C FL C L, 
and 

(4.13) FL D FVL = pL D pL. 

Hence L is admissible. 
Next, we have 

(4.14) LD LD FL, 

where all inclusions have index 1 . But L is non-special and hence L is special. □ 

The above proof in fact shows the following. Suppose that L e X with FL = 
p • L x . Then L(£) x = pL(£) . If FL X = pL , then L(f) x = L{£) . 

Thus to any distinguished L we have associated a projective line F(L/FL) 
and a family of admissible special lattices parametrized by the F -points of this 
projective line. These projective lines have a natural ¥ p 2 -structure which we now 
describe. 

For any W -lattice L in C , we have 

(4.15) FL = VL F 2 L = FVL = pL p~ 1 F 2 L = L. 

Lemma 4.5. Suppose that p~ 1 F 2 L = L , and let 

L = { x e L | p~ 1 F 2 x = x }. 
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Then Lq is a Z p 2 -module and 

Lo ®z p2 W ~ L. □ 

If L G X is distinguished, then L is preserved by the a 2 -linear endomorphism 
p~ 1 F 2 , and we have L ~ Lq <S>z 2 W • Moreover, FX is also preserved by p~ 1 F 2 , 
and (FL) = F(L ) . Thus, the two dimensional F -vector space L/FL has a 
natural F p 2 -structure: 

(4.16) L/FL ~ L /FL ® Fp2 F. 

We may then view any line £ as an element of ¥(L /FL )(¥) . We denote by P^ 
the projective line F(L Q /FL Q ) over F p 2 . 

Lemma 4.6. Under the isomorphism 

L/FL ~ Lo/FLo ® Fp2 F, 

t/ie automorphism induced by p~ 1 F 2 on L/FL coincides with l®a 2 on 
Lq/FLq <g> Fp2 F . Hence, 

p-'F'W)) = L(a 2 (i)\ 
where t is identified with a point in P^(F) . □ 

Corollary 4.7. A lattice L(£) associated to a distinguished L is super special, i.e., 
has a(L(£)) = 2 , if and only if £ e P^F^) . 

Proposition 4.8. Suppose that L e X with a(L) = 1 , and let 

L = F~ 1 (FL + VL) . 
Then L is distinguished and L = L(£) for a unique line I G P^(F) \ P^(F p 2) . 

Proof. Let L ± = c ■ L . Then 

(FL) X = (FL) 1 - n (VL) 1 - = V^L 1 - n F _1 L X = p~ x c ■ (FL n VL). 

On the other hand, F 2 L = F 2 L + pL . Let S = L/pL , and let / and v be 
the a -linear resp. cr _1 -linear endomorphisms of S induced by F and V. Since 
FV = VF = p , we have fv = vf = and so ker(/) = im(v) and ker(v) = im(/) 
are 2-dimensional subspaces of S . However, for any L G X there is some j > 2 
with F J L C pL and hence / is nilpotent. If f 2 = , then F 2 L = since 
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both lattices have index 4 in I and this would imply a(L) = 2 , contrary to our 
assumption. Therefore, since im(/) is 2-dimensional we must have that im(/ 2 ) is 
one-dimensional and im(/ 2 ) = im(/) fl im(i>) . Hence 

F 2 L = F 2 L+pL = FLHVL . 

It follows that F(FL) = p-c~ 1 (FL) ± . On the other hand, FL is admissible, since 

pFL = p(FL + VL) CpLc F 2 L = FL fl VL C FL C FL = FL + VL 

where all inclusions are of index 1. It follows that FL e X and hence also L e X . 
Finally L = L(£) for the line 

£ = L/FL C L/FL . □ 

We summarize the above construction in the following theorem. 
Theorem 4.9. There is a natural G'(Q P ) -equivariant map 

Lex 

which induces a bijection 



]\(F L (¥)\F L (¥ p ,))^X\X 



The map associates to (L,£) , where £ C L/FL is a line, the element L = L(£) e 
X . 

The action of g G G'(Q P ) on the index set of the left hand side is lifted in the 
obvious way to the whole set appearing on the left hand side. 

Remark 4.10. It can be shown that the map above is in fact a morphism, i.e., is 
the map on F -points induced by a morphism of schemes over Spec ¥ p , 

]J P - — > m ss . 

Lex 

This can be shown by the method of Oort, [24], or using Cartier theory, as in 
Stamm, [30]. Using either of these methods one can construct a morphism of 
schemes over Spec ¥ p , 



G'(Q)\ {]lV L )xG'(A*)/K* 
_ Lex 

which turns out to be the normalization of the curve M. t 



The 'distinguished curves' cross at the superspecial points. To describe this, it 
will be useful to have a normal form for superspecial lattices. 
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Lemma 4.11. Fix 5 G Z* 2 with 5 a = —5 . Let L G Xq be super special and 
standard. 

(i) Suppose that L = L . Then there is a basis e\ , e 2 , e% , e^ for L over W 
such that es = Fe\ , = Fe 2 , Fe^ = pe± , Fe^ = pe 2 and such that the matrix 
for the polarization is 

(< ei,ej >)ij = 6- 

(ii) If L= pL 1 - , then L = FL' where V G X with V = (L') ± . 

Proposition 4.12. Suppose that L G X is superspecial and standard. 

(i) If L 1 - = L, consider lattices L such that Ld Ld FL and such that FL = 

pL 1 - . Such L 's are distinguished; there are p+1 of them, and they can be described 
explicitly as follows. Let e\,. . . ,e4 be a standard basis as in Lemma Then 
the distinguished L 's have the form 

L = W(e t + /je 2 )+FL 

where [i G Z^ 2 such that n\i a = —1 mod p . 

(ii) If L = pL 1 - , then the distinguished L 's containing L with index 1 are those 
associated, as in (i), to V = F~ X L . 

Proof of Lemma J^.ll. Since p~ 1 F 2 is a a 2 -linear automorphism of L , we can 
write L = Lq ®i 2 W for the rank 4 lattice Lq of fixed points of p~ 1 F 2 . Let 
So = Lq/pLq , a 4 -dimensional symplectic vector space over ¥ p 2 , and note that 
FLq/pLq is an isotropic 2-plane in , which is paired with the quotient L Q /FL . 
We can then choose e\ and e 2 G Lq whose images form a basis for Lq/FLq and 
such that < ei,e2 >= 0, after modification by elements of FL , if necesssary. 
The elements e\ , e 2 , es := Fe\ and e^ := Fe 2 then give a W -basis for L , and 
Fes = F 2 e\ = pe\ , and Fe^ = F 2 e 2 = pe 2 , as required, since e\ and e 2 G L . 
The matrix for the polarization is then 

where A =< e, Fe >, with e = ( 61 | . 

- - - \e 2 J 

Note that det(A) G Z^ 2 , and that 

(4.18) - l A a =< Fe, e > a =< e, Ve >=< e, Fe >= A, 

since V = pF~ l and so, on L , V = F 2 ■ F -1 = F . If we change the vector e 
to a • e, for a G GL 2 (Z p 2) , then A changes to aA t a cr . Since det(A) G Z^ 2 and 
since the norm map iV : Z^ 2 — > Z* is surjective, it is easy to check that, for a 
suitable choice of a we can obtain aA t a cr = 8 • 1 2 ■ □ 




w {-'a i) 
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Proof of Proposition 4-12. Let us prove (i). Using the standard basis of Lemma 4.11, 
we have L = [ei, e 2 , e^, e 4 ] (the square brackets indicate the IU-span) and FL = 
\pe\,pe2, e3, e 4 ] . Any lattice L with L D L D FL and with [L : L] = 1 has the 
form 

(4.19) L = W ■ (aei + 6e 2 ) + FL, 

where at least one of a and b G W is a unit. If a is a unit, we can write 
L = [ei + fj,e 2 ,pe 2 , e 3 , e 4 ] . Then 

(4.20) FL = [e 3 + ^ <T e 4 ,pe 4 ,pei,pe 2 ] and pL^ = [e 4 - ^e3,pe 4 ,pei,pe 2 ]. 

Comparing, we see that \i must be a unit and that fifi a = — 1 mod p , as claimed. 
It is easy to check that the case in which a is not a unit yields no solutions. The 
assertion (ii) is trivial. □ 



Corollary 4.13. The map appearing in Theorem 4-9. is surjective. Any lattice in 
Xq has p + 1 preimages which all lie on distinct lines. In fact, the preimages of 
L G Xq correspond to the distinguished lattices F~ 1 L where L ranges over the 
lattices associated to L in (i) of Proposition 4-12 ( resp. to distinguished lattices 
L associated to L in (ii) of Proposition 4-12). Finally, the images of two distinct 
lines and P^, have at most one lattice in common which then lies in X . 



Proof. The last assertion follows since, if L, V G X , L ^ V , both lie on P^ , 
then L = L + L' . □ 



The next result gives a standard basis for a distinguished lattice. 



Lemma 4.14. Let L be a distinguished lattice which is standard. 

(i) If FL = pL 1 - , then there exists a W -basis e±, . . . , e 4 of L such that = Fe\ , 

e 4 = Fe 2 , Fes = P e i > Fe^ = pe 2 , and such that the polarization has matrix 



(< e t ,ej >) itj = 6 



( 1 

-1 

V 



-p 



p 



(ii) If FL 1 - = pL , then L = FL' where V G X with FL' = p ■ L 



Proof of (i). Let L be the fixed points of p~ 1 F 2 on L. Since FL = pL 1 - , < , > 
induces a nondegenerate symplectic form on the two dimensional ¥ p 2 -vector space 
L /FLq . Choose e\ and e 2 G L whose images in L /FL are a basis for this 
space and such that < ei, e 2 >= 5 . Let e 3 = Fei and e 4 = Fe 2 , so that, as in 
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Lemma 4.11, Fes = F 2 e\ = pe\ and Fe^ = F 2 e2 = v e i • The polarization then 
has matrix 



In the present case, however, A = mod p . A Hensel's Lemma argument shows 
that we can replace e by ae + bFe with a G GL 2 (Z p 2) and b G M 2 (Z p 2) to 
achieve A = , while preserving the condition < e, e>=5J. □ 

Recall that G'(Q P ) , given by (4.6) above, acts on the set of admissible lattices. 
For any lattice L, (gL) 1 - = v(g)~ 1 g(L- L ) . If L G X is a special lattice, with 
L = c ■ L 1 - , then gL = v(g)c ■ (gL) 1 - , so that gL is again special. Moreover, 
a(gL) = a(L) so that the subset of superspecial lattice is preserved. Also, if L 
is distinguished, and if g G G'(Q P ) , then gL is again distinguished. Since the 
valuation of v(g) is an arbitrary integer, for any L G X (resp. L G X ) there 
is g G G'(Q P ) such that gL (resp. ) is standard with (gL) 1 - = gL (resp. 
F(gL) = p ■ (gL) 1 - ). By Lemmas 4.11 and 4.14, we have: 

Corollary 4.15. G'(Q P ) acts transitively on the set of superspecial lattices and 
on the set of distinguished lattices. 

We would finally like to compute the stablizers in G'(Q P ) of the superspecial 
and distinguished lattices. 

Let B' be as above, and, identifying Q p 2 with a subfield of B' p , write B' p = 
Q p 2 + IKQ)p2 for an element IT G B p > x with IT 2 = p and such that Ila = a a Ii , 
for a G Q p 2 . Let Cq be the fixed set for the automorphism p~ 1 F 2 of C , and let 
II operate on £ by F . By construction, IT 2 = p , and so £ is naturally a left 
vector space over B' p of dimension 2 . 

Lemma 4.16. Let 



(4.21) 




where 




and A =< e, Fe >= — t A a , as in the proof of Lemma 4.11. 



End K (C,F) := {a G End K (£) | Fa = aF }. 



Then, 



End,e(£,F) = End Qp2 (£ , F) = End s; (£ )- 



The polarization on £ induces a Q p 2 -bilinear symplectic form on £ ; which 
still satisfies < Fx, y >=< x, Vy > a . 
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Lemma 4.17. Let U be a left B' p -vector space with a B' p -Hermitian form ( , ) : 
U x U — > Bp . 77ms (6x, cy) = 6(x, y)c L and (y, x) = (x, y) L , where b i— > is i/ie 
main involution on B' p . Write 

(x,y) = (x,y) 5 + (x,y)i<JII, 

where (x,y)o and (x,y)i G Q p 2 . Then, 

( , )i:f/xf/^Q p2 

is a symplectic Q p 2 -bilinear form on the Q p 2 -vector space U such that 

(*) (ILt^H^IW, 

and 

(x,y) = -(x,Uy) 1 . 

The map ( , ) i— > ( , )i yields a bisection between the space of B' p -Hermitian forms 
on U and the space of symplectic forms satisfying (*). Moreover, 

G'(Q P ) = {ge GL{C) \< gx, gy >= u(g) < x, y > and Fg = gF} 
GLb'(Cq) I (gx,gy) = v(g)(x,y) }. 

Proof. We just check the behavior of IT . We have 

(4.22) (Tlx, y) = U(x, y) 5 + U(x, y^SU 

= -p(x,y)iS-(x,y)^6U 

and 

(4.23) (x,Uy) = -{x,y) SU-p{x,y) 1 8 . 
Thus 

(4.24) (Ux,y) 1 = -(x,y)^ = (x,Uy) i , 

as required. It is at this point that the factor of 5 is required in the formulas. □ 

Let O' = Ob 1 = ZL2 +IIZ„2 be the maximal order in B' . If L is an admissible 
lattice such that p~ x F 2 L = L , then the fixed point set L of p~ 1 F 2 is naturally 
an O' -lattice in the B' p -vector space Co , and dim^ L /nL = 2 . Conversely, 
given any O' -lattice A with this last property, we set F = a® IT on L = L(A) : = 
W<S>z p2 A • Then since IT 2 = p on A , we have L D FL D pL and dim F L/FL = 2 , 
i.e., L is admissible, and p~ 1 F 2 L = L . The following is easily checked, using the 
formulas of Lemma 4.17. 
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Lemma 4.18. (i) Suppose that L G Xq is superspecial with L = L , and let 
€!,...,€£ be a standard basis as in Lemma Then e[ = 5~ 1 ei and e' 2 = 5~ 1 e2 

is an O' -basis for L , and the matrix for the B' p -Hermitian form on Co is 

((eX-)k,- = 1 2 • _ 

(ii) Suppose that L G X is distinguished and that FL = pL 1 - , and let ei, . . . ,€4 
be a standard basis as in Lemma 4-H- Then e[ = 5~ 1 ei and e' 2 = —5~ 1 e2 form 
an O' -basis for Lq and 

((4e;))ij = (_ n n )- 

Thus, in classical language, cf. [29], [8], the superspecial lattices correspond to 
local components of the principal genus of quaternion Hermitian lattices, while the 
distinguished lattices correspond to local components of a non-principal genus of 
such lattices. 

In less classical language we may describe our results in terms of the Bruhat- 
Tits building of the adjoint group G' ad over Q p , comp. also [11]. The building 
B(G' ad , Q p ) is a tree and may be identified with the fixed points 

B{G'^%)=B{G' ad X) F ■ 

The special vertices in B(G' ad , Q p ) correspond to the equivalence classes of lattices 
L C C which are F -invariant. Here two lattices L\ and L 2 are equivalent if 
L\ is homothetic to L 2 or to L 2 ■ Hence the special vertices are in one-to-one 
correspondence with the distinguished lattices L which are standard and with 
FL = pL 1 - . The non-special vertices in B(G' ad , Q p ) correspond to the edges in 
B(G' ad ,K.) whose vertices are interchanged by F . Equivalently, they correspond 
to pairs {L, FL} of lattices in Xq which are standard. We thus obtain bijections 

I«Zx {special vertices in B(G' ad , Q p )} 

and 

Xq hZx {non-special vertices in B(G' ad ,Q p )} . 

These bijections are G'(Q P ) -equivariant, where g G G'(Q P ) acts on the Z -component 
on the right via n 1— > n + ord(is(g)) . The action of F on the left corresponds to 
the translation n 1— > n + 1 on the first factor and the trivial action on the second 
factor on the right. Furthermore, a lattice L e X and L e X are incident (i.e. 
L G P^) if and only if the corresponding vertices of B(G' ad , Q p ) lie on one and the 
same edge. 
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In these terms the stabilizer K d of a distinguished lattice L e X is a special 
maximal compact subgroup of G'(Q P ) , and the stabilizer K ss of a superspecial 
lattice L e X is a non-special maximal compact subgroup of G'(Q P ) . 

Remark 4.19 We return, for a moment, to the global situation, and recall that 
X is the set of distinguished lattices in C . As observed in Remark 4.10, our 
calculations 'show' that the supersingular locus M ss is a union of rational curves 
and that the irreducible components are in bijection with the set 



(4.25) G'(Q)\(X x G(A P )/K p j ~ G' (Q)\(G' (Q p ) / K* x G(A P )/K P 

where is the stabilizer in G'(Q P ) of a fixed distinguished lattice LeX. These 
curves cross, p + 1 at a time, at the superspecial points, and there are p 2 + 1 such 
crossing points on each component. The set of all crossing points is in bijection 
with the set 



(4.26) G'(Q)\\X x G(K P )/K p j ~ G'(Q)\\G'(Q P )/K? x G(A P )/K P 

where K* s is the stabilizer in G'(Q P ) of a fixed superspecial lattice L e X . 

We finally observe two consequences of our description of M ss . 

Fix a factorization -D(-B) = DiD 2 , and let K = \\ t Kt be the compact open 
subgroup of G(Af) with local factors 

r Kf if £ i Di, 

*Q = < Kf S£l\D 2 , 

Here, we have fixed a maximal order R in £? , and for £ f D(B) , we fix an 
isomorphism M 2 (B £ ) ~ M 4 (Q^>) such that M 2 (R e ) ~ M 4 (Z^) . Then let K° = 
G(Q^) n M 4 (Z f ) . Thus, for £ | D 1 (resp. f | D 2 ), K t is the stabilizer of a 
Hermitian -lattice of principal (resp. non-principal) type, and, for I \ D(B) , 
Kg is a hyperspecial maximal compact subgroup of G(Qg) . Note that, in contrast 
to the general assumptions above, K is not neat. Still, for a fixed prime p \ D(B) , 
one can consider the coarse moduli space M.k (the quotient by a finite group of 
one of the schemes considered above) and its points over F. Let B^ denote 
the definite quaternion algebra with D(B^) = D{B)p . Then, by (4.25), the 
components of the supersingular locus in the fiber of Mr at p correspond to the 
classes of maximal Hermitian lattices in the genus of type (Di,pD 2 ) for B^ . 
An explicit formula for this number H{Di,pD 2 ) was found by Hashimoto and 
Ibukiyama [9]. In the case D(B) = 1 , so that B = M 2 (Q) , the abelian varieties 
parameterized by Mk(^) have the form A ~ Aq , where Aq is a principally 
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polarized abelian surface. Thus, in this case, M.k — ^2,1 , and the description of 
the supersingular locus reduces to some of the information given by Katsura and 
Oort [12], Theorem 5.7, and Ibukiyama, Katsura and Oort [10]. In particular, the 
number of irreducible components of the supersingular locus is H(l,p) . 

As another example, fix a square free positive integer D and distinct primes p\ 
and P2 relatively prime to D . Consider indefinite quaternion algebras B\ and 
B 2 over Q with discriminants D(B\) = Dpi and D(B 2 ) = Dp 2 . Let G\ and 
G 2 be the associated groups, via (1.3). As in (4.5), let G[ be the twist of G\ 
at p 2 and let G' 2 be the twist of G 2 at p\ . These groups are both associated 
to the definite quaternion algebra b[ P2 ^ ~ B2 , and are isomorphic. Fix an 
isomorphism G[ ~ G' 2 and compatible isomorphisms 

Gi(A^ lP2 ) ~ G[(A p f lP2 ) ~ G' 2 (Af P2 ) ~ G 2 (Af P2 ), 

and let K PlP2 = K PlP2 = K PlP2 be a sufficiently small compact open subgroup. 
Also let 

K ljPl =K*l, for *! = d or ss, 



pi 

K l,P2 = K l 2 



K 2 , P1 = K pi , 

K 2jP2 =K*^ for *i = d or ss, 
where the notation is as above. Let 

k; 2 =K^K 2 , P1 K 2 , P2 . 

Let M*i = M. K *i and M 2 2 = M K *2 be the corresponding moduli schemes, 
defined over Z( P2 ) and %( Pl ) respectively. 

Then, using (4.25) and (4.26), there are (non-canonical but equivariant) bisec- 
tions between various sets of irreducible components or crossing points as follows: 



Components \^(M 2 x F pi ) s - s J ~ Components [{M^ x F P2 ) S ' 
Components (^(Mf x F pi ) s - S -^ ~ Crossing points (^(Mf x F P2 ) S - S -^ 
Crossing points ( (M 2 x F Pl ) s ' s - J ~ Components ( (Mf x F P2 ) S ' S - J, 



Crossing points y(Mf x F Pl ) s - s J ~ Crossing points y(Mf x F P2 ) S ' 

Here we have written (Mf x F P2 ) S S - for the supersingular locus of the fiber over 
p 2 of M*i , where *i = ss , for example. These results are in the spirit of those of 
Ribet [25] , [26] , who considers components and their crossing points for the fibers 
of Shimura curves and modular curves at primes of bad reduction. 
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§5. Endomorphism algebras and points of proper intersection. 

In this section, we consider the points of intersection of the special cycles in the 
supersingular locus, using the information obtained in section 4 about the structure 
of this locus. In particular, in the decomposition 

Z(d 1 ,u 1 ) x M . . . x M Z(d r ,u> r ) = |J Z(T,u>) 

T G Sym 4 (Z (p) )> 
diag(T) = (di,... ,d r ) 

of (3.6), we fix a matrix T and we obtain a criterion, in terms of T , for Z{T, u) 
to consist of isolated points. We also show that, even when det(T) ^ , there can 
be components of the supersingular locus in the image of Z(T, to) in M ss . 

We retain the notation of sections 2-4, and we begin by obtaining information 
about the endomorphism rings of various types of admissible lattices. 

For an admissible lattice L, let Ol = End^/(L,F) be the Z p -algebra of W - 
linear endomorphisms of L which commute with F. Note that Endw(L, F) is an 
order in the Q p -algebra End/c(A F) = C' p = C Q p ~ M 2 (B' p ) . Also, observe 
that End w (L, F) = End w (F j L,F) for any j G Z. If L = c ■ L 1 - is special, we 
have 

(5.1) (F J L) = p J c ■ (F J L) 1 - . 

Thus, to determine Ol for L E X we may assume L = L 1 - . 

By Lemma 4.18, we immediately have the following. 

Lemma 5.1. For any superspecial lattice L e X$ , (resp. any distinguished lattice 
L G X ) Endiy(L, F) (resp. Endw(L, F) ) is a maximal order in C' p . 

In either case, this order is isomorphic to M^iO'} , where O' = Z p 2 + ilZ p 2 , as 
in section 4. The map M^iO'} — > M2(¥ p 2) given by reduction modulo IT can be 
described as follows. Consider the case of L G X . As in section 4, let L be the 
fixed points of p~ 1 F 2 on L . Then define 

(5.2) red L : End w (L, F) — > End Fp2 (L /FL ) ~ M 2 (F p2 ) 
as the composition 

(5.3) End w (L, F) End Zp2 (L , F) — > End Fp2 (L /FL ). 
This map is surjective. The surjective reduction map for L G X 

(5.4) red z : End w (L, F) — > End Fp2 (L /FL ) ~ M 2 (F p2 ) 
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is defined analogously. Note that L/FL ~ Lq/FLq ®f 2 ^ ? an d that the endomor- 
phism a induced on L/FL by a G Endjy(L, F) is red^(a) <8> 1 . 

Next, suppose that L E X \ Xq , and let L be the unique distinguished lattice 
associated to L by Proposition 4.8. Recall that FL = FL + VL . In particular, 
for every element a G Ervdw{L, F) , aFL C FL , so that aL C L , and there is a 
natural homomorphism which is injective, 

(5.5) End w (L,F) ^End w (L,F). 

On the other hand, there is a unique line £ C L/FL such that L = L(£) is the 
inverse image of I in L . 

Lemma 5.2. Let \ X . With the notations introduced above, 

End w (L, F) = { a G End w (L, F) | a(£) C £ }. 

Here a is the endomorphism of L/FL induced by a . In fact, there are two 
possibilities. 

(i) If £e P £ (F) - P z (F p4 ) , then 

End w (L,F) = (red z )- 1 (F p2 -l 2 ). 

(ii) If £e P^(F P 4) - P L (¥ p 2) , then 

End M /(L,F) = (red L )- 1 (F p4 ), 
for some embedding F p4 <^-> M 2 (F p 2) . 

Proof. As remarked above, the automorphism of L/FL = Lq/FLq ®w p2 F induced 
by p~ 1 F 2 is just 1 <g> a 2 . Since, for any a G Endjy(L, F) , a commutes with 
this automorphism, 6t{£) C £ implies that a(a 2 (£)) C cr 2 (£) . Since a non-scalar 
endomorphism can have at most two eigenlines, a(£) C £ and cr 4 (£) ^ £ implies 
that a = a • 1 2 , for a G F p 2 . If a 4 (£) = £ but o~ 2 (£) ^ £ , and if a is not a 
scalar endomorphism, then £ and o~ 2 (£) are the distinct eigenlines of a . Then 
F p 2[o:] ~ F p 4 , and any endomorphism (3 , with (3 G Endw(L, F) must lie in this 
subfield of M 2 (F p2 ) . □ 

Note that the lattices in (ii) of lemma 5.2 are characterized intrinsically by the 
condition that F 4 L = p 2 L but F 2 L ^ pL . We let X^ be the set of lattices 
appearing in (ii) and X^ = X \ X^ \ X the set appearing in (i). Recall that 
O l = End w (L, F) and O l = End w (L, F) . Then 

red L (0 L ) = red L (End M /(L, F)) ~ M 2 (F p2 ) if L G X 

(5.6) red L (0 L ) = red z (End w (L, F)) ~ F p4 if L G 
red z (C L ) = red z (End M /(L, F)) ~ F p2 if L G 
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In particular, the endomorphism algebras of all the L 's with L G and with 
a given associated L coincide. Any endomorphism of one such L preserves all 
lattices L' e X in the image of . 

Recall that C' p = End;c(£, F) , and let 

(5.7) Vp = { x G C' p | x* = x, and tr°(x) = }. 

Note that Ol and O l are invariant under the involution * on C' p . Indeed, let 
L L = cL and x e O l . Then 

x*(L x ) C L 1 - , i.e. x*(L) C L . 

Similarly, if FL = cL 1 - and a; G O l , then a;*(L x ) C V- , i.e. i*(FL) C FL , i.e., 
x*(L) C L , since commutes with F. 

For L e X and for Lei, let 

(5.8) iV L = End M /(L, F) n V£ and N L = End w (L, F) n V£. 

These are Z p -lattices in Vp on which the quadratic form given by squaring, x 2 = 
q(x) ■ id is valued in Z p . 

We now describe the reduction maps for distinguished and for superspecial lat- 
tices. We start with the case of distinguished lattices. 

Lemma 5.3. Let L G X , and put = red^iV^) . Then is equal to 

{x = a ■ I2; a G F p 2, a a = —a} 

and the ¥ p -valued quadratic form q on is given by x 2 = q(x) ■ I2 , i.e. q(x) = 
—a-a a . In particular, q does not represent 1 and hence the Clifford algebra C(n^) 
is isomorphic to ¥ p 2 . The following diagram is commutative 

N L -U Z p 

red 4 I 




Proof. Replacing L by F J L we may assume that L is standard with FL=p-L 
The symplectic form < , > on C induces a nondegenerate alternating pairing 



(5.9) 



< , >: L/FL x L/FL — >F 
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This pairing descends to a non-degenerate alternating F p 2 -bilinear pairing on 
Lq/FLq with values in F p 2 . The induced involution on EndF p2 (Lq/FLq) is com- 
patible with the reduction map, 

reduce*) = redi(x)* , x G Ol ■ 

Now any endomorphism x of the 2-dimensional symplectic vector space Lq/FLq 
over F p 2 with x* = x is a scalar. Hence for x G we get 

red (a?) = a ■ I2 , a G F p 2 . 

But x G acts on Lq/pLq preserving the subspace FL /pL . Since x com- 
mutes with F , it acts on the subspace as of • ±2 . The condition tr°(x) = implies 
therefore that a = —a a . Therefore we have proved that is contained in the 
subspace above. It is easy to see that we have in fact an equality. The remaining 
assertions are obvious. □ 

Next we consider the case of superspecial lattices. 

Lemma 5.4. Let L G Xq and put xvl = red^A^) . 
(i) xil is isomorphic to 

{xe M 2 (F p2 ) I t x a = x and tr(x) = } 




a G F p , be F p2 }. 



The Fp -valued quadratic form q on is given by x 2 = q(x) ■ I2 , i.e., q(x) = 
-(a 2 + bb a ) . 

(ii) Let C{xvl) be the Clifford algebra of the three dimensional quadratic space xil ■ 
Then the natural map C(xii) M 2 (F p 2) is an isomorphism. 
(Hi) The following diagram is commutative. 

N L -U Zp 
n L -U Fp . 

Proof. Replacing L by F^L we may assume L ± = L . On Lq/FLq we have the 
non-degenerate anti-hermitian form 

(5.10) ( , ) : Lq/FLq X Lq/FLq — ► Fp2 

induced by the formula 



(5.11) 



(v, w) = (v, Fw) mod p , 
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where v and w are representatives of v and w in L$ . We may find a basis of 
Lq/FLq such that the induced involution on M2(F p 2) is given by x ^ t x a . Now 
the lemma is proved in a way similar to Lemma 5.3. above. □ 

We now return to the points of intersection of the special cycles in the super- 
singular locus. Let T G Sym 4 (Z( p )) with detT 7^ and lo C V{K p ^) a with 
corresponding special cycle Z(T,u) . Let £ G Z(T,u>) correspond to the collec- 
tion (A%, l, A, ?) p ; j) . By Corollary 4.3, the point corresponding to the collection 
A,7p) lies in M ss (¥) . Thus, End°(A ? ,t)°P = = C ~ M 2 (B') , where 
is the definite quaternion algebra over Q with discriminant D(B)p . The last 
isomorphism here can be chosen so that the Rosati involution corresponds to the 
involution xhx' = t x t of M 2 (B') . Then, as in (3.8.vi), 

(5.12) = 7'~{ie M 2 (B') I x' = a; and tr(x) = }. 

The components ji, ■ ■ ■ ,j& of j lie in V , and therefore, in particular, we must 
have T > if Z(T,u) is to be non-empty. 

Let L be the contravariant Dieudonne module of the formal group Ao(p) , where 
we write A^(p) ~ Aq(p) 4 , as in section 4. By choosing an isogeny of £ with the 
chosen base point £ D we obtain, as in section 4, an identification C = L ®vk K> 
of its isocrystal with that of the base point. Then Lei, and there is a natural 
algebra homomorphism 

(5.13) C 6 ® z % P = End(% 0° p ®z Z p End w (L, F) = O l C C' p . 

Let iV^ = Endw(L, F)C\Vp , as in (5.8) above. The collections of endomorphisms 
j induce collections of elements of Endjy(£, F) and of Vp , which we will denote by 
the same letters. Let M be the Z p -submodule of N L spanned by the components 
jii J21 jz-, H °f J • We have the following commutative diagram: 

{j' 1 ,...,j 4 } C Q — ► End w (L,F) D N L D M 

(5.14) I I I I 

v c a — > c v d v; 

Recall that T = G Sym 4 (Z( p )) C Sym 4 (Z p ) is the matrix of inner products of 
the elements ji, ■ ■ ■ ,j& with respect to the quadratic form on Vp . Thus we have 
the following basic observation: 

Lemma 5.5. At a point of intersection £ G Z(T,u) fl .M SS (F) with corresponding 
lattice L G X , the matrix T^ = T is represented by the lattice Nl = Endw(L, F)D 
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Vp in the quadratic space Vp . In fact, T is the matrix for the restriction of the 
quadratic form on AT L to the sublattice M spanned by ji, ■ ■ ■ , j± ■ 

Suppose L e X \ X Q with associated distinguished lattice L . Recall that 
Ol C Oi and let Om be the Z p -subalgebra of O^ = Endw(L, F) generated 
by ji,..., 34, i-e., by M . Also let C(M) be the Clifford algebra of M . Let 
n L = redi(N L ) and let m = = red^(M) , so that 

M C N L C N L 
(5-15) | I | 

C riL C n^. 



Lemma 5.6. Suppose that L e X \ X with associated distinguished lattice L 

(i) The natural map C(M) — > Om is an isomorphism. 

(ii) There is a commutative diagram 

O m C O l 

I I 
red L (0 M ) red L (0 L )~M 2 (W p2 ) 

II u 
C(m L ) ^ C(n z )=F p2 -l 2 , 



Proof. The inclusion of C(m^) into C(n^) is induced by the inclusion of quadratic 
spaces m C . We obviously have a commutative diagram with surjective vertical 
arrows, 

C(M) — O m 
(5.16) | I 

C(m) — red z (0 M ). 

But the upper horizontal arrow is surjective since both algebras are generated by 
M . This proves that the lower horizontal arrow is surjective. By the statement at 
the beginning it is also injective which proves the equality sign at the south-west 
corner of the diagram in (ii). The rest of the Lemma follows from Lemma 5.3. □ 

Next let us consider the case when L e Xq . We use somewhat similar notation: 
let n L = red L (N L ) and m = m L = red L (0 M ) ■ 

The same arguments yield: 



44 



Lemma 5.7. Suppose that L e Xo is super special. There is a commutative dia- 
gram: 

O m C O l 
I I 
red L (G M ) C red L (0 L ) ~M 2 (F p2 ) 

C(m L ) C(n L ) 

Our next task will be to show that the matrix T modp in M 4 (F p ) controls the 
size of m . More precisely, we now list the possibilities for m in the non-superspecial 
and the superspecial case separately. 

Lemma 5.8. Let L e X \ Xq with associated L e X . The possibilities for 
m = m^ are the following: 

(i) If diniF p m = 1 , then T has rank 1 modulo p and does not represent 1. 

(ii) // m = , then p\T . 

Proof. The first alternative corresponds to the case where m = , by Lemma 5.3. 
The assertion now follows from Lemma 5.5. □ 

Lemma 5.9. Let L e X Q . The possibilities for m = mt and C(m) C M 2 (F p 2) 
are the following: 

(i) The rank of T mod p is 3, or equivalently dimm = 3 . Then C(m) ~ 
M 2 (F p2 ) . 

(ii) The rank of T mod p is 2. Then dimm = 2 and C(m) ~ M 2 (F p ) . 

(hi) The rank of T mod p is 1 and dimm = 2 . Then m is of the form 
m = mo + t where r is the radical and dim mo = dimr = 1 . In this case 

C(m) ~ C(m r[e]/(e 2 ) 

where 

c(mo) = I W r , 

and the element e acts on C(mo) by the nontrivial automorphism of order 
2 . 

(iv) The rank of T mod p is 1 and dimm = 1 . Then 

F P ©F P 
F p2 



C(m) 

(v) T = mod p and dimm = 1 . Then 

C(m)=A(m)^F p [e]/(e 2 ) . 

(vi) m = . Then T = mod p and C(m) = ¥ p is in the center of M 2 (F p 2) 
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In cases (iii) resp. (iv), mo resp. m is a nondegenerate line, so that the quadratic 
form on it is isomorphic to either x 2 or ax 2 , with a G F* \F* ' 2 , yielding a Clifford 
algebra F p © F p or F p 2 . 

Lemma 5.10. In cases (iii) and (iv) above, when an ¥ p 2 arises in the Clifford 
algebra C(m) , this ¥ p 2 is not central in M 2 (F p 2) . 

Proof. Choose x G m spanning a nondegenerate line for which C(F p a:) ~ F p 2 . 
Then x is an endomorphism of the 2 -dimensional F p 2 -vector space Lq / FLq with 
ti(x) = 0, and with x 2 = q(x) ■ id where q(x) ^ F p ,2 . This last condition is 
equivalent to our hypothesis on the Clifford algebra. Thus, x has two distinct 
eigenvalues ±y/q(x) on Lq/FLq , and hence does not lie in the center. □ 

We can now describe the intersections of our special cycle with the supersingular 
locus. 

Theorem 5.11. Suppose that £ G Z(T,u) with image in the supersingular locus 
Ai ss (¥) with corresponding L G X . 

(i) The rank of T = modulo p is at most 3 . 

(ii) If represents 1, then L G X and £ is a point of proper intersection. 

(iii) If L G X\X Q , with associated distinguished lattice L , the whole distinguished 
associated to L in the supersingular locus, and passing through £ , occurs in 

Z(T,u) . In particular, £ is not a point of proper intersection. 

Proof. The reduction of T = modulo p is the matrix for the quadratic form 
on the images of ji, . . . , 34 in m = , resp. m = , and m has dimension at 
most 3 . This proves (i). If L G X \ X , then T = does not represent 1 , by 
Lemma 5.8. Furthermore in this case by Lemma 5.3. 

C(m) C F p2 • 1 c red £ (£>z/) , 

for any V G , which is not superspecial. This implies M C Om C Ol> ■ If now 
V G P^ is superspecial it follows that M C Om C Ol> by specialization. □ 

It remains to consider the cases where T does not represent 1 . We first treat 
the case when p\T . 

Theorem 5.12. Suppose that p \ T and that £ G Z(T,uj) has image in M ss (¥) 
with corresponding L G X . Then £ is not a point of proper intersection. More 
precisely: 

(i) If m = redi(M) = then each of the p + 1 distinguished P 1 's through pr(£) G 
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A4 SS occurs in the image of Z(T,u) , i.e., for every distinguished L with L D L D 
FL , we have M c Endjy (L, F) ; furthermore red^(M) = . 
(ii) If m = redi(M) is a null line in , then there is a unique distinguished L 
with L D L D FL and with M C Endjy(L, F) ; furthermore red^(M) = . Hence 
there is a unique distinguished P 1 passing through pr(£) G .M ss and contained in 
the image ofZ(T,u>) . 

Proof. We may assume that L = L . First suppose that m is a null line, and 
choose xq G M such that xq = red.L(xo) spans m = redi,(M) . The endomorphism 
xq of the two dimensional vector space L / FL satisfies x\ = but x$ ^ . Thus 
im(xo) is a line in Lq/FLo . 

Lemma 5.13. Assume that L = L 1 - . The lattice L defined by FL = xq{L) + FL 
lies in X and L eF L . Moreover, M C End w (L,F) , and red^(M) = . 

Proof. Since xq commutes with F, we clearly have F(FL) = xq(FL) + F 2 L = 
x (FL) +pL C FL. Similarly one sees that V(FL) C FL , i.e., pFL C F(FL) , 
hence FL is admissible. Note that (FL) 1 - D L ± = L D FL with [(FL) 1 - : L] = 
[L : FL] = 1 . 

To show that FL is distinguished, it will suffice to prove that F(FL) C p(FL) 1 - , 
i.e., that < F 2 L, FI>CpW. But 

<F 2 L,FL> = <x (FL)+pL,x (L)+FL> 
C < x (FL),x (L) > +pW 

(5.17) = < FL,x 2 (L) > +pW 

C < FL, FL > +pW 

C pw: 

Here we have used the fact that Xq = x and that x\ = , i.e., that Xq(L) C FL . 
We conclude that FL G X and hence also Lei. 

Next, we must show that every element of M preserves L or, equivalently, 
FM . In fact, we show that M ■ L C FL , so that red^(M) = . First consider 
the reduction sequence 

(5.18) — > M — >M W p ■ x — >0, 
where 



(5.19) 



M = { y G M | y(L) C FL }. 
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It suffices to prove the inclusions xq(L) C FL and y(L) C FL for all y G Mo . 
Recall that, for x G M , x 2 = (/(a;) • id . Since p | , the resulting quadratic form 
on M/pM is identically zero, and so C(M/pM) = A(M/pM) . In particular, for 
any x\ and x 2 G M , = —£2^1 mod p , i.e., 

(5.20) xix 2 (L) C x 2 xi(L) + pL. 
Now, for y e M , 

y(FL)=yx (L)+y(FL) 

(5.21) Cx y(L)+pL + F(y(L)) 

C x (FL) + F 2 L 

C F(x (L)+FL) = F 2 L. 

Next, observe that = q(xo)-id and q(xo) = mod p implies that Xq(L) C pL , 
not just FL . Thus 

x (FL) = x 2 (L)+Fx (F) 

(5.22) CpL + Fx (L) 

= F(FL + x (L)) = F 2 L. 

This completes the proof of the Lemma. □ 



To finish the proof of (ii) , we show that the distinguished lattice FL constructed 
in Lemma 5.13 is unique. Note that ker(x ) = im(x ) . If L' = W ■ u + FL is 
another distinguished lattice, whose image £' = V /FL is distinct from ker(x ) , 
then 

(5.23) xo(£')=im(x )^£', 
so that L' is not preserved by xq . 

Now suppose that red L (M) = , i.e., that M ■ L C FL . Let FL = W -u + FL 
be any distinguished lattice with L D FL D FL . We want to show that, for any 
x G M , x(L) C FL = pL 1 - or, equivalently x{FL) C F 2 L = p • (FL)- 1 . But now 

(5.24) <x(FL),FL> = < Wx(u) + Fx(L),Wu + FL > 

C W < x(u),u > +pW. 

But now, since x* = x , 

(5.25) < x(u), u > = < u, x(u) > = — < x(u), u > 

so that < x(u),u >= 0. Thus < x(FL),FL > C pW , i.e., x(FL) C pF(L) ± = 
F 2 L , as required. This concludes the proof of Theorem 5.12. □ □ 



We now turn to the case when p\T but T does not represent 1 
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Theorem 5.14. Suppose that p \T and that T does not represent 1 . Let £ G 
Z(T,u) with pr(£) G A4 SS (¥) and with corresponding L G X . Then £ is not a 
point of proper intersection. More precisely: 

(i) If diniF p m = 1 and m does not represent 1 , then precisely two of the p + 1 
distinguished P^ 's through pr(£) occur in the image of Z(T,u) . These are the 
only two distinguished lattices Li and Li with Li D L D FLi and with M C 
End w (Li, F) (i = 1,2) . Furthermore red L% (M) ^ (0) , i = 1,2 . 
(n,) If diniF p m = 2 and m = mo + t as m Lemma 5.9, (Hi), where m does not 
represent 1 , i/ien precisely one of the p + 1 distinguished P^ 's through pr(£) 
occurs in pr(Z(T,u>)) . It is the only distinguished lattice L\ with L\ D L D FLi 
and witfi M C End w (Li,F) . Furthermore red^M) 7^ (0) . 

Proof. We may again assume L 1 - = L . We first consider case (i). Choose xq G M 
such that = red^^o) spans m = redi(M) . Then x is an automorphism of 
Lq/FLq with 

Xq = £ • 1 , 

where £ G F* \ F* ' 2 . Furthermore, by Lemma 5.8., x is not central. Hence x 
has two distinct eigenvalues £1 = y/e and e 2 = —\ft in F p 2 . Let £1 and E 2 be 
the corresponding eigenspaces and let FL\ and FL 2 be the corresponding lattices 
in C , 

FL c C L , i = 1, 2 . 

Since x commutes with F and F , the eigenspaces Ei and E 2 are preserved 
by F and V , hence FLi and FL 2 are admissible. To see that FL\ and FL 2 
are distinguished, it suffices to see that FLi C p • Lj- , i.e., that 

(FLi, Li) Cp- W , i = 1,2 . 

Equivalently we have to see that the eigenspaces E\ and E 2 are isotropic with 
respect to the antihermitian form (5.11) on L /FL . If v G Fj , then xof = £i • v 
and 

£ • (t>, u) = (£V, V) = (XqU, u) = (x l>, X V) = (£iV, EiV) = -£■ (v,v). 

It follows that FLi and FL 2 are distinguished. 

The lattices Li = F~ 1 (FL i ) G X for i = 1 and 2 are the distinguished lattices 
appearing in the statement of (i). We have red^ (M) 7^ since xq induces 
an automorphism of the eigenspace Fj . On the other hand any y G M with 
redi,(n) = , i.e., with y(L) C FL , also satisfies y(Li) C L C Li . It follows that 
M C End w (Li, F) , hence (i). 

Now we consider case (ii). Let x G m with Xq = £ • 1 , for £ G F* \ F* ' 2 , and 
let y be a generator of the radical r . Then x~oy = —y Q xo . Therefore y maps 
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the eigenspace E\ of xq in L /FL to the eigenspace E 2 and the eigenspace 
E 2 to E\ . Since y q = , but y 7^ , precisely one of the two eigenspaces is 
annihilated by y . The corresponding lattice is distinguished and yields as in case 
(i) the lattice Li appearing in the statement of (ii). □ 

Corollary 5.15. Let £ G Z{T,uj) C Z{di,uJi) x M ■■■ x M Z{d r ,uj r ), where di G 
Sym n . (Q)>o with ri\ + • • • + n r = 4 , cf. (3.1) and (3.6). Then £ is a point of 
proper intersection if and only if its fundamental matrix T = is non-singular 
and represents 1 over Z p . In this case £ is super singular and super special. 

A topic we have not touched upon in the present paper is to describe the shape of 
the intersection of our cycles in the case of improper intersection, or, equivalently, 
to describe, for T G Sym 4 (Q) >0 , the cycle Z(T, u) when its dimension is positive. 
We refer to the companion paper [21] to the present one for more information on 
this topic. 
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§6. Intersection multiplicities. 

In this section we consider the intersection multiplicity at a point of proper 
intersection. More precisely we return to the setup of the third section, i.e., we fix a 
decomposition 4 = ni+- ■ -+n r , where rii > 1 for all i , elements di G Sym n . (Q)>o 
and Ui C V(A^) ni giving rise to special cycles Z(di, ui), . . . , Z(d r , ui r ) . We 
fix a point £ G Z(di,ui) Xm ... x^vi Z(d r ,u> r ) with det(T^) 7^ and where 
T = represents 1 over Z v . Let £ correspond to (A, t, A, ff ; j l5 . . . ,j r ) . Since 
det(T) 7^ and since T represents 1 over Z p , the associated Dieudonne module L 
is superspecial and corresponds to a formal group A of dimension 2 and height 4, 
with a collection of endomorphisms j = (ji, • • • , J4) spanning a Z p -submodule M 
of rank 4 in Endw(L, F) . By changing the trivialization of the rational Dieudonne 
module we may assume that L = L 1 - , i.e., that A is equipped with a principal 
quasi-polarization . An immediate application of the theorem of Serre and Tate 
shows that the formal completions at £ of M. and of its closed subschemes may 
be interpreted as versal deformation spaces, i.e., with obvious notation, 

(6.1) Mt = Be£(A,\ A ) 

(6.2) Z(di,Ui)z =Def(A,\ A ;ji) 

(6.3) Z(T, uj)^ = (Z(di,oj 1 ) x M ■ ■ ■ x M Z(d r , u r ))^ 

= Def(A A^;j) = Def(A X A ;M). 

Here uj = u± x ■ ■ ■ x u r . Recall that any x G M satisfies x* = x and tr(x) = , 
that the quadratic form on M is given by x 2 = q(x) - id , and that T is the matrix 
for that quadratic form with respect to the basis ji, . . . , j 4 . Since T represents 
1 over Z p , there exists an x G M such that x\ = id . The quadratic lattice M 
can be written as M = Z p ■ xq + M , where M = Xq . Moreover, if x G M , then 
xxo = —xqx , since the subalgebra of End^(L, F) generated by M is the image 
of C(M) , the Clifford algebra of M . 

The idempotents e\ = ^(1 + xq) , ei = |(1 — xo) - recall that p 7^ 2 - give 
a splitting A — A\ x A2 , with Ai = e\A and A2 = Z2A of dimension 1 and 
height 2 . If x G Mq , xe± = e2X , and so Mq can be viewed as a submodule of 
Hom(Ai, A2) ■ Let Li = eiL be the Dieudonne module of Ai . Then L = L\@L2 ■ 
Furthermore L\ and L2 are paired trivially under the symplectic pairing on L . 
Indeed, if v\ G L\ and v 2 G L 2 , we have 

< vi, V2 > = < eivi, e 2 v 2 > = < vi,eie 2 v 2 > = , 

since e* = e\ . It follows that < , > induces a perfect symplectic pairing < , >j 
on Li , i.e., A\ and A2 are equipped with principal quasi-polarizations. Since 
a principal quasi-polarization on a p -divisible formal group of dimension 1 and 
height 2 deforms automatically we obtain a natural identification 

Def (A, X A ; M) = Def (^1, A 2 ; M ) . 
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The length e(£) of the local Artin ring appearing on the right was determined by 
Gross and Keating in section 5 of [7]. Since we have assumed in all of the above 
that p 7^ 2 , we may as well continue to make this assumption, although Gross and 
Keating do not. Choose a basis tpi, tp2, ^3 f° r -^o such that 

qiu^i + u 2 ip2 + u 3 ip 3 ) = tip ai u\ + e 2 p a2 uj + e 3 p as u\, 

with < ai < ci2 < a 3 . Thus, over 7L V , T is equivalent to the diagonal matrix 
diag(l, t!p ai , e 2 p a2 , e 3 p a3 ) . 

Proposition 6.1. (Gross, Keating, [7], Proposition 5.4) If a\ + ci2 is even, then 
e (0 = e p(^?) i s equal to: 

oi-l (oi+o 2 -2)/2 

J2(i + l)(a 1 +a 2 + a 3 - 3i)p l + ( a i + !) ( 2a i + a 2 + a 3 - 4i)p* 

i=0 i=ai 

+ ^(a 1 + l)(a 3 -a 2 + l)^ ai+a2 )/ 2 . 
7/ ai + a 2 is odd, t/ien e(£) = e p (T^) is equal to: 

oi-l (oi+o 2 -l)/2 

^(i + l)(ai + a 2 + a 3 -3iy + ^ (ai + l)(2ai + a 2 + a 3 - 4i)p* . 

i=0 i=ai 

Corollary 6.2. TTie cycles Z(di,u>i), . . . ,Z(d r ,u> r ) intersect transver sally at the 
point £ if and only if ord(det T%) = 1 . 

Proof. The above formulas show that e(£) = 1 if and only if ord(detT^) = 1 . If 
this is the case, the cycles Z(di, uii) have to be irreducible and reduced locally at 
£ , and the intersection multiplicity in the sense of Serre, which is bounded by the 
length, is equal to 1. In this case the cycles Z(di, Ui) are all regular at £ and their 
tangent spaces give a direct sum decomposition of the tangent space of M. at £ . 
For all this, cf. [5], Prop. 8.2, and Example 8.2.1. 

At this point we have completely answered the question a) at the end of section 3. 
What is not clear is whether the length e(£) is indeed the intersection multiplicity 
of Z(d\, ui), . . . , Z(d r , uj r ) at £ . This is the content of the question b) of section 
3. 

Conjecture 6.3. Let £ be an isolated intersection point of Z(di,u>i), .. . ,Z(d r ,uj r ) . 
Then 

L L 

(O z{duLOl) (8) ... (8) Oz(dr, Ur )h = iPz(d ltUl ) ® • • • ® O z{dr ^ r) ) i , 
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hence e(£) is the intersection multiplicity of Z(di,ui), . . . , Z(d r ,u> r ) at £ . 

We stress that this conjecture is reasonable only because M. is smooth over 
SpecZ(p) . Indeed, Genestier [6] has shown that in the Drinfeld-Cherednik sit- 
uation of bad reduction the analogues of the special cycles considered here may 
have embedded components. On the other hand, assume in our situation that 
Z(di,0Ji) is an intersection of ni divisors in M. . Then if £ is an isolated inter- 
section point of Z(d\,uji), . . . ,Z(d r ,u} r ) it follows that each partial intersection 
Z{d ix , c^ij fl . . . fl Z[di s , u>i s ) ( 1 < %\ < . . . < i s < r ) is locally at £ a complete in- 
tersection. Hence it also follows that the length e(£) is the intersection multiplicity 
of Z(d±, u>i), . . . , Z(d r , u r ) at £ , and the above conjecture holds true. 

Remark 6.4. Assume that £ G Z{d\,u)\) fl ■•• fl Z(d r ,uj r ) is a point with 
fundamental matrix T = which is non-singular and represents over Z p a unit 
e E Z* \Z*' 2 . Therefore there exists xi E M such that x\ = e ■ id . Hence we 
obtain an action of Z p 2 = Z p [y/e\ on A , 

a : Z p 2 — > End(A) . 

We may write M in the form M = Z p ■ x\ + Mi , where Mi = x^ . For x G Mi 
we have xx\ = —xix . Comparing with the definitions in the companion paper 
to this one, we see that (A, a) is precisely one of the formal groups with Z p 2 - 
action considered there [] and that the elements of Mi are special endomorphisms 
in the sense of that paper. In particular, the formal completion of Z(di,u>i) fl 
• • • fl Z{d r , u r ) at £ coincides with the formal completion of the corresponding 
subvariety of the Hilbert-Blumenthal surface considered in [21]. 
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§7. The total contribution of isolated points. 

In this section we will consider the total contribution of the points of proper in- 
tersection of our special cycles. Using our previous results and a counting argument, 
we are able to give an explicit formula. 

We return to the global situation of sections 1 and 2 and fix data as follows. We 
assume as always that p\2D(B) and that K = K p ■ K p where K p is the standard 
maximal compact subgroup ( see the end of section 4), and where K p is neat. We 
then have the moduli scheme M. = M. KP which is smooth over SpecZ( p ) . As 
in section 3, we fix n\, . . . , n r with 1 < rii < 4 and with ri\ + . . . + n r = 4 . 
For i = 1,... , r, choose positive definite matrices di G Sym n . (Z( p ))>o and K p - 
invariant open compact subsets u>i C V{K p ^) ni . We then have the cycles Z(di,uJi) , 
i = 1, . . . , r . We then define the contribution of the points of proper intersection 
to the intersection number of Z(d\, u>i), . . . , Z(d r , u> r ) to be 

(7.1) <Z(d 1 ,u 1 ),...,Z(d r ,u r )>^ :=$>(0 • 

Here the sum runs over the points of proper intersection £ in Z(d\, ui\) Xyvi • • • Xm 
Z(d r , u r ) , and e(£) denotes the length of the local ring at £ , as described in section 
6. Note that, if Conjecture 6.3 were known to hold, this is also the local intersection 
multiplicity at £ . 

In the special case r = 1 , we let d\ = T , and we have the cycle Z(T,uj) , whose 
image in M. lies in the supersingular locus M ss . Then Z{T, uj) is a collection of 
isolated points if and only if T represents 1 over Z p (Corollary 5.15). In this case 
we use the notation 

(7.2) <Z(T^)> P = J2 e ^)' 



In general, by (3.6) and the analysis of the previous sections, we may write 
(7.3) < Z(d u wi), . . . , Z(d r , u r ) >P roper =J2< Z ^ u ) >p> 

T 

where the summation is over T e Sym 4 (Z( p ))>o which are nonsingular, represent 
1 over Z p , and have diagonal blocks d\, . . . ,d r : 

(d x ... \ 

d 2 ... 

T = 

\ ... d r ) 
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We will now give more explicit expressions for the above entities. For this it 
will suffice to give an expression for (7.2). But the results of section 6 show that 
the intersection multiplicities e(£) in the sum of (7.2) only depend on T and even 
only on its Z p -equivalence class. As in Proposition 6.1, we denote this integer by 
e p (T) and thus may write 

(7.4) < Z(T,u) > p = e p (T) ■ \Z(T,u)(¥)\ . 
It remains to determine the cardinality of Z(T, uj)(¥) . 

As before, let B' be the definite quaternion algebra with discriminant D(B)p , 
let C = M 2 (B') , and let V = {x G C \ x' = x and tr(x) = } . Let G' be 
as in (4.5). Recall that we also have fixed an isomorphism G'(A P ) ~ G(A P ) , and 
a base point £ Q = (A , t Q , A D , rj p ) G .M SS (F) such that the associated Dieudonne 
module L a G X is superspecial, with stabilizer K' p in G'(Q P ) . Then, under the 
parametrization (4.7), the set of superspecial points in .M SS (F) corresponds to the 
double coset space 

(7.5) G'(Q)\(g'(Q p )/K> p x G(Aj)/ff*), 

cf. Corollary 4.15. For a superspecial point (A, i, A, ff) of A1 SS (F) , the choice an 
isogeny 7 : (A,t) — > (A ,t ) compatible with the polarizations determines a pair 
(g p ,g p ) e G'(Q P )/K' p x G(A p f )/K p , and the passage to G'(Q) -orbits removes the 
dependence on the choice of 7 . 

The choice of an isogeny 7 also yields an identification of the space End (A, l)° p 
with End (A o , t ) op = C , and of the space of special endomorphisms of (A, l, A) 
with V(Q) . Let Q' T (Q) C V'(Q) 4 be the fibre over T of the map defined by the 
quadratic form on V'(Q) , 

(7.6) V'(Q) 4 — Sym 4 (Q) . 



Returning to the set Z(T, u)(¥) , we consider the map 

(7.7) Z(T,oj)(¥) ^ G'(Q)\(q' t (Q) x G'(Q P )/K' p x G(A p f )/K^ 

defined as follows. To a point ^ = (A, t, X,ff;j) G Z(T,u>)(¥) , and a choice of 
isogeny 7 : (A, t) — > (A G , t ) , there is an associated triple (7^, g p , g p ) , where 
7*j G V (Q) 4 is the 4-tuple of endomorphisms determined by j and 7 . Again, the 
passage to G'(Q) -orbits removes the dependence on the choice of 7 . 

It is not difficult to describe the image of Z{T,uj){¥) . For y G O^(Q) , the 
triple (y, g p , g p ) lies in the image if and only if 

(i) The images of the components of y under the inclusion V Endw(£, F) 
preserve the lattice g p L , and 

(ii) The image of y under rf lies in g p ■ uj . 
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We note that the condition (i) is equivalent to the assertion that the components 
of the 4-tuple g~ 1 y lie in 

(7.8) V'(Z P ) = V'(Q P ) n End w (L, F) = N L . 

We let (p' p be the characteristic function of V'(Z p ) 4 , let (f p , = char(u;) be the 
characteristic function of u , and set ip'f = <p' p <S> <fi P f ■ Then ip'j G S(V (Af) 4 ) K ' . 
Conditions (i) and (ii) can then be summarized as follows. 

Lemma 7.1. The G'(Q) -orbit of the triple (y, g p , g p ) lies in the image of Z(T, cu)(F) 
if and only if <p' f {g~ l y) ^ , where g = (g p , g p ) G G'(A f ) . 

Note that the function (y, g) \— > <p'f(g~ 1 y) is invariant under the diagonal action 
of G'(Q) on the left and under the action of K' = K' p K p and of Z'(A f ) on the 
right. The total contribution of the superspecial points may be expressed as an 
integral. 

Theorem 7.2. Let T e Sym 4 (Z( p )) >0 be non-singular and such that T repre- 
sents 1 over 7h p . Let ui C F(Aj) 4 be K p -invariant open and compact, and let 
K' = K p K p C G'(A f ) . Let pr(K') be the image of K' in Z'(A f )\G'(A f ) ~ 
SO(V')(A f ) . Then 

<Z{T,u) > p =e p (T)-vol( W (K'))- 1 -I TJ (<p' f ) . 

Here <p'f = ip' p ®ip p G 5 , (l / (A/) 4 ) as above, and lTj(<fi'f) denotes the theta integral 

G'(Q)Z'(A f )\G'(A f ) y en T(Q) 

The measure dg is induced by an arbitrary Haar measure on Z'(Af)\G'(Af) 
and the atomic measure on Z'(Q)\G'(Q) . The coefficient e p (T) is given by the 
formulas in Proposition 6.1. The identity of the Theorem remains valid if T is 
nonsingular but not positive definite, since, in that case, T is not represented by 
V , and hence both sides of the identity vanish. 

Proof. By Lemma 7.1, we see that 

(7.9) |z(z>)(F)|= £ <^GrV). 

G'(Q)\(n' T (Q)xG'(A f )/K'Z'(A f )) 

On the other hand, since pr(K') is neat, the stabilizer in Z'(Q)\G'(Q) of a coset 
gK'Z'(Af)/Z'(A f ) is trivial. Thus, we have 

(7.10) \Z(T,oo)(¥)\=vol( W (K'))- 1 J £ V/GrV)^, 

G'(Q)Z'(A f )\G'(A f ) y en r(Q) 
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for a measure as described in the Theorem. In combination with (7.4), this gives 
the claimed expression. □ 

Corollary 7.3. In the situation of the beginning of this section, 

< Z(d 1 ,u 1 ),...,Z(d r ,u r ) >^ = ]Te p (T) vol(pr(K')) _1 -ItjWs) • 

T 

where T runs over all T G Sym 4 (Z(- p - ) ) > o which represent 1 over 7L V and have 
diagonal blocks d±, . . . ,d r . The function ip'f = <p' p <8> G ^(^'(A/) 4 ) is defined 
by 

V?; = char V'{Z p f 

ip P j = char(o;i x . . . x u> r ). 

Remark 7.4. Formula (7.10) expresses the quantity \Z(T,uj)(W)\ as a product of 
orbital integrals. More precisely, note that the components of y G Q' T (Q) span 
a 4 -dimensional subspace of the 5 -dimensional space V . Since G' acts on V 
via its projection to SO(V) , the stabilizer of y in G'(Q) is precisely Z'(Q) , the 
kernel of this projection. Since G'(Q) acts transitively on fl' T (Q) , we can unfold 
to obtain: 

(7.11) 

|Z(2»(F)| = voi(K')- 1 / v/GrV) <fo 

= vol(K')- 1 vol(Z'(Q)\Z'(A / )) O y (^) O y (^), 

for orbital integrals 

(7.12) o y ( v p )= [ rf{g- l y)dg, 

and 

(7-13) O y {<p' p )=f ^(g^dg. 

JZ'(Q P )\G'(Q P ) 

In our main theorem (in section 9), we will identify the right hand sides of the 
formulas of Theorem 7.2 and Corollary 7.3 as special values of derivatives of Fourier 
coefficients of certain Eisenstein series. In the next section we will explain more 
precisely the Eisenstein series in question. 
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§8. Fourier coefficients of Siegel Eisenstein series. 

In this section, we recall, from [19], the construction of certain incoherent Siegel 
Eisenstein series and the structure of the Fourier coefficients of their derivative at 
s = , the center of symmetry. To be more precise, these Eisenstein series occur 
on the metaplectic cover of the symplectic group of rank 4 over Q , and have an 
odd functional equation. Their Fourier coefficients are parameterized by rational 
symmetric matrices T G Sym^Q) . In [19], a formula was given for the derivative 
at s = of such a coefficient, when det(T) ^ . 

We retain the notation of section 1, and we refer to sections 1 - 6 of [19] for more 
details. Thus B is an indefinite quaternion algebra over Q of discriminant D(B) , 
C = M2(B) , V is given by (1.1), and G is given by (1.3), etc.. In particular, V 
is a five-dimensional quadratic space over Q with signature (3, 2) . Let x = Xv 
be the quadratic character of A x /Q x attached to V : x( x ) = ( x i det(V))A , where 
( , )a is the global Hilbert symbol. Note that Xoo(— 1) = 1 • 

Let W be a symplectic vector space of dimension 8 over Q , with a fixed 
symplectic basis e\, . . . , e^, e[, . . . , e 4 , and let be the metaplectic extension of 
Sp(Wa) , with Siegel parabolic Pa • For s G C and for x as above, let I^s.x) be 
the global degenerate principal series representation of Ha ■ As explained in [19], 
(2.9), the representation 1^(0, x) has a direct sum decomposition into two types of 
irreducible representations. One of these types are the irreducible summands, like 
114(1/) , associated to five-dimensional quadratic spaces with character xv ■ The 
other type are the irreducible summands associated to incoherent collections, 
in the sense of section 2 of [19]. One such summand is 114(C) , associated to the 
incoherent collection C , defined as follows. For any finite prime I , Cg = Ve , while 
Coo = V^o , where is the quadratic space over R of signature (5,0) . There is 
a surjective map 

(8.1) X f : S((C A/ ) 4 ) = S(V(A f ) 4 ) — 114(C)/ C h(0, x)f- 

A section <E>(s) G h(s, x) is standard if its restriction to the standard maximal 

compact subgroup K H in H A is independent of s. For ipf e S(V(Af) 4: ) , let 

<&/(s) be the standard section of l4(s,x)f such that <&/(0) = \(<Pf) ■ Let <&(s) = 

5 5 
3><x>(s)<8>3>/(s) , where $<^(s) is the standard section of /4(s,x)oo whose restriction 

to K Hoc is the character det 5 . Then $(s) is an incoherent section with $(0) G 

n 4 (C) . The incoherent Eisenstein series 

(8.2) E(h,s,$)= 

-rePq\HQ 

converges for Re(s) > | , and its analytic continuation vanishes at the point s = , 
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[19]. There is a Fourier expansion 

(8.3) E(h,s,$)= E Ah,s,^), 

TeSym 4 (Q) 

with respect to the unipotent radical of P . When $(s) = <S>£^i(s) is a factorizable 
section, and when det(T) ^ , there is a product formula 

(8.4) E T (h, s, $) = H W T ,e(he, s, 

i<oo 

where Wr,<(^,s,$f) is the local generalized Whittaker integral, cf. section 4 of 
[19]. For fixed h, T, and $ , there is a finite set of places S such that, [19], 
Proposition 4.1, 

(8.5) J] W T ,t(he, s, = C s (2s + 4)- 1 ( s (2s + 2)~\ 
and hence 

(8.6) E T (h, s, $) = ( s (2s + 4)- 1 ( s (2s + 2) 1 J] W Ttt (h e , s, $ e ). 

ees 

Since det(T) ^ , the factors Wr/ihi, s, $^) have an entire analytic continuation. 

Fix T with det(T) ^ 0. Since E T (h,0,$) = , at least one of the factors in 
the product formula (8.6) vanishes at s = . In particular, by Proposition 1.4 of 
[19], the factor at £ vanishes whenever the five-dimensional quadratic space Ci 
does not represent T . Let Diff (T, C) / be the set of finite places at which Ct fails 
to represent T , and let 

, ^ , s f Diff(T,C) f U {oo} if sig(T) = (3,1) or (1,3) 

(8.7) Diff (T,C)= ' ' 11 1 J 1 J 

[ Dift(T, C)/ otherwise. 

By Corollary 5.3 of [19], |Diff(T,C)| is odd; and, by Corollary 5.4 of loc. cit., 

(8.8) ord Er(h,8,Q) > |Diff(T,C)|. 

s=0 

Thus, the only nonsingular T for which E' T {h, 0, $) can be nonzero are those for 
which |Diff(T,C)| = 1 . We will relate the value E' T (h,to,$) for Diff (T,C) = {p} 
to the numbers < Z(T, uj) > p in the previous section. 

Let us fix a finite prime p . We wish to give a formula for E' T (h, 0, $) if T e 
Sym 4 (Q) is nonsingular with Diff (T,C) = {p} . Let B' be the definite quaternion 
algebra over Q which is ramified at p and whose invariants coincide with those of 
B at all finite primes other than p . Let C = M2 (-£?') , and let 

(8.9) V = { x e M 2 (B') I x = x and tr(x) = }, 
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with quadratic form defined by squaring, as in section 1. Let G' = GSpin(V) be 
defined by the analogue of (1.3). Note that there is an exact sequence 

(8.10) 1 — >Z' — > G' — ► SO(V) — > 1 

of algebraic groups over Q , where Z' is the center of G' . 

We fix identifications B'(A p f ) = B(A P ) , and hence V'(A p f ) = V(A P ) , and 
G'(A P ) = G(A P ) . We also assume that (pf G S(V(Af) 4 ) is factorizable, so that 
<p f = <p p ® cp p , and we can view <p p as a Schwartz function on V'(A P ) 4 . Recall 
that there is a surjective map 

(8.11) X' f : S(V'(Af) 4 ) — > U,(V') f C h(0, X )f. 



Recall, [31], [19], that the local degenerate principal series representation 1^(0, Xp) 
has a direct sum decomposition with irreducible factors 

(8.12) 7 4iP (0, x P ) = Ra(V p ) © R 4 (V;). 

Let T G Sym 4 (Q) be nonsingular with Diff(T, C) = {p} . Then the linear func- 
tional 

(8.13) W T , p (h,0,-) :/ 4>P (0,x P ) — C 

vanishes identically on R±{V P ) = Ra{C p ) , and does not vanish identically on the 
summand R^Vp) , [19], Proposition 1.4. We choose a standard section & p (s) , 
with $p(0) G R^iVp) , and such that for a given h G /7q p , 

(8.14) ^(^,0,^)^0. 

Let G ^((V^) 4 ) be a Schwartz function whose image X p (<fi p ) in /^(O, x P ) is 
$p(0) . Note that V is positive definite, and let ip'^ G ^((V^) 4 ) be the Gaussian, 
f'ooix) = exp (—Tvtr(q(x))) . Finally, let <p'j = <p' p <8> </?j so that 

(8.15) ^ = <p'oo ® V/ = V4, ® ^ ® ^ e S(V'(A) 4 ). 



Recall that the metaplectic group 77a acts on the space 5 , (V A/ (A) 4 ) via the Weil 
representation us = , defined using our fixed additive character ip of A/Q . For 
g G G'(A) and h <E H A , let 



(8.16) 
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be the theta function attached to <p' , and let 

(8.17) I(h, V ') = lf 6(g,h,ev(<P'))d9, 

A JG'(Q)Z(A)\G'(A) 

for the Tamagawa measure dg on Z(A)\G'(A) , and where ev(<//) denotes the 
projection of ip' to the subspace of functions all of whose local components are 
even, cf. [19], (7.19). Note that 6(g, h, ip') can be defined by the same formula for 
g G 0(V')(A) , and that 

(8.18) I(h,<p') = [ 0{jg,h,tp')dg, 

JO(V')(Q)\0(V')(A) 

where vol(0(V')(Q)\0(V')(A), dg ) = 1 . 
For g G G'(A) and h G , let 

(8.19) 9 T ( 9l h ;( p')= J2 WMGrV), 

yeii' T (Q) 

and 

(8.20) e TJ (g,<p' f )= ^/GrV), 

yen> T (Q) 

where 0Tj(g, <p'f) depends only on gf . Recall that <p' = ip'^ <8> cp'^ , with 

(8.21) ip'M = e -^«^)), 

the Gaussian attached to V . This function is invariant under G'(M) and, for 
y G Oy(Q) , it has the value 

(8.22) ¥4(y) = e- 27rMT) . 
Therefore, for h G , we have 

(8.23) e T (h,g;<p') = £ (u;(/*y) GrV) 

yen' T (Q) 

where yo is any fixed element of Q' T (Q) . 
For h G ifoo , and for y G Cl' T (Q) , set 

(8.24) W*(fc):=(w(fcyj(y ). 
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More explicitly, as in (11.74) of [19], if h has lwasawa decomposition h = (n(b)m(a)k,t) G 
Sp 4 (R) x C 1 ~ Mp^W^) , for b G Sym 4 (R) , a G GL 4 (R) + , and k G K Hoo , then 

(8.25) wj(ft) = * • det(a)' e(tr(Tb)) e -^ aTa) det(A; )| 

= * • det(a)* e(£r(Tr)) det(fc)*, 

where r = 6 + za*a . 

Recalling that Z'(R)\G'(R) ~ SO(F')( R ) is compact, we have the following 
formula for the T -th Fourier coefficient of the theta integral: 

(8.26) 

2I T {h, ip') 

= / T (g,h;ev(ip'))dg 

JG'(Q)Z'(A)\G'(A) 

= W*(h)- [ e T , f (9,ev(<p'f))dg 

JG'(Q)Z'(A)\G'(A) 

= W*(h) ■ vol(SO(y')(K),doo0) • / QT,f(gM<P'f))dfg, 

JG'(Q)Z'(A f )\G'(A f ) 

where d/g is the measure arising from the counting measure on Z'(Q)\G'(Q) and 
the Haar measure on Z'(Af)\G'(Af) ~ SO(V')(Af) coming from some choice of a 
gauge form /j, on SO(V) . Also d^g is the Haar measure on 5 , 0(V)(M) induced 
by \i. 

With the notation just described, and for h G iioo , Corollary 6.3 of [19] spe- 
cializes to 

WL(e,0,Q p ) 

^ E ^ ^= wZ { e,0,4 ' 2lTih ' 9) ' 
if T is nonsingular with Diff (T,C) = {p} . 

Substituting the expression (8.26) for the Fourier coefficient of the theta integral 
found above, we obtain: 

5 

Proposition 8.1. Suppose that <E>(s) = ^So(s) ® $/(s) wif/i $/(0) = \f(<Pf) , is 
an incoherent standard section. For h G -f/oo , and for each T G Sym^Q) with 
det(T) ^ and Diff(T,C) = {p} , choose ip' p and $' p (s) , such that W T , p (e,0, & p ) ^ 
0. Then 

, /0 WL(e,Q,$ v ) 
E' T (K 0, <&) = vo\(SO(V')(R)) ■ W± /2 (h) ■ W ^0,<S,>) ' lT >fW- 
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Here 



lT,f{<p'f) = I ev W(g V) d f g 

JG'(Q)Z'(A f )\G'(A f ) yGn , T(Q) 

= / 6T,f(g,ev(<p' f )) d f g, 

JG'(Q)Z'(A f )\G'(A f ) 

and the measures are as described after (8.26) above. 
If the function (p'j is locally even, then the integral 

(8.28) lT,f(<Pf)= [ e TJ (g,<p' f )d f g 

JG'(Q)Z'(A f )\G'(A f ) 

occurs in Theorem 7.2, where the measure arises from an arbitrary Haar measure 
on Z'(Af)\G'(Af) , and the quantity 

(8.29) volMif))- 1 ^,/^) 

is independent of the choice. Therefore, we can obtain the expression 



(8.30) E^(h,0,$) = vo\(SO(V')(R)pr(K')) -W} /2 (h) 



where the factor vol(5'0(V / )(]R)pr(i^ / )) is computed using the Tamagawa measure 
on SO(V')(A) . Hence, since pr{K') is neat, 

vo\(SO(V')(R)pr(K')) = 2\SO(V')(A) : SO(V')(Q)SO(V) (R)pr(K')\-\ 

and the quantities in (8.30) separated by a dot do not depend on any choice of 
measure. 
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§9. The main theorem. 

In this section we assemble the results of previous sections and state our main 
results. 

We begin by further specializing the formula of Proposition 8.1. Specifically, we 
need more information about the factor 

1 ' > W T , r (e,0,%y 

Fix the prime p with p \ 2D(B) , and assume that ip p is the characteristic function 
of V(Z p ) 4 . Recall that $ p (s) is the standard section with $ p (0) = X p (ip p ) . Also, 
let (p' be the characteristic function of the lattice V (Z p ) 4 , and let <&' p (s) be the 
standard section with <& p (0) = A p (<^ p ) ■ 

Recall that a nonsingular T e Sym 4 (Q p ) is represented by precisely one of the 
quadratic spaces V(Q P ) and V'(Q P ) , [19], Proposition 1.3. 



Proposition 9.1. Suppose that <p p , <p' p , $ p , $ p are as above, and that T e Sym4(Q p ) 
with det(T) ^ . 

(%> If W^ p (e, 0, $ p ) ^ , then T e Sym 4 (Z p ) . 

(ii) If T e Sym 4 (Z p ) and if T is represented by V'(Q P ) , then W T , P (e,0,<S> p ) ^ 
. 

(Hi) If T e Sym 4 (Z p ) is represented by V'(Q P ) , and if T represents 1 , then 
where e p (T) is the local intersection multiplicity given in Proposition 6.1. 



The proof will be given in section 10. 



A subset uj C V(A P j-) n is said to be locally centrally symmetric if it is 
invariant under the action of the group ^(A^) . The characteristic function (p u e 
S(V(A P j) n ) of such a set is locally even, as in (8.17), i.e. tp w = ev(<^ w ) . The 
function ip'^ = <p' p ® ip u G S(V (Af) n ) is then locally even as well, so that the 
expression (8.30) holds for the derivative of the Fourier coefficients of the associated 
Eisenstein series. 



Our first main result is the following. 



Theorem 9.2. Assume that p \ 2D(B) and that cp p , (p' p , <E> P , $ p are as above. 
Let uj C V(A p f ) 4: be a locally centrally symmetric K p -invariant compact open 
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subset. Let <3>(s) = $00 (s) <S> $>p( s ) ® ^ > /( s ) be the standard section corresponding 
to Lp = ifoo ® (f p ® (pK G ^(^^(A) 4 ) iwii/i </?j = char(c<;) , c/ Lemma 7.1. Suppose 
that T G Sym 4 (Q) with det(T) ^ and with Diff(T, C) = {p} . 

(i) If T ^ Sym 4 (Z( p )) >0 , then Z(T, u) = , < Z(T, to) > p = , and 

E^(h,0,^) = 0. 

(ii) If T G Sym 4 (Z( p ))>o represents 1 over Z p , t/ien Z(T, u;) is zero dimensional, 
and, for h G , 

E' T {h, 0, $) = \vo\{SO{V'){R)) ■ Wr /2 (h) • vol(pr(K)) • logp < Z(T, u) > p . 

Note that, if T G Sym 4 (Z( p )) > o does not represent 1 , then ^(T, contains 
components of the supersingular locus (Corollary 5.15 and Theorems 5.12 and 5.14). 
In this case, we do not have a formula for < Z(T, u>) > p . 

In Theorem 9.2, the chosen gauge form /j on SO(V) = Z'\G' determines 
the Haar measure on SO(V')(R) used to compute vol(SO(V')(R)) . The cor- 
responding gauge form on the inner twist SO(V) = Z\G determines the mea- 
sure on Z'(Af)\G'(Af) used to compute vol(pr(K)) . Note that the product 
vol(S , 0(y / )(IR)vol(pr(iir)) is independent of the choice of \i . 

Proof of Theorem 9.2. Beginning with formula (8.30), and using (iii) of Proposi- 
tion 9.1 and Theorem 7.2, we have 

E' T (h, 0, $) = vo\(SO(V) (R)pr(K')) ■ w} /2 (h)- 
W'(e,0,$ p ) , 1 

= vol(SO(V')(R)pr(K')) ■ W^ /2 (h)- 

■ hogp ■ (p 2 + l)(p - 1) • Cp(T) • vol(pr(K'))"^T,/(^/) 
= vo\(SO(V) (R)pr(K')) ■ W^ /2 (h)- 

■ ^logp ■ (p 2 + l)(p - 1)- < Z(T, u) > p . 

To finish the proof, we simply note the following relation between volumes. 

Lemma 9.3. Recall that K p = GL 2 (0 Bp )r}G(Q p ) and K' p = GL 2 (0 B > p )r}G' \Q P ) . 
Then, for the Haar measures on Z'(A f )\G'(A f ) , Z'(Q P )\G'(Q P ) , Z(A f )\G(A f ) , 
and Z(Q p )\G(Qp) determined by the fixed gauge form fx and the corresponding 
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form on the inner twist, 

vol(pr(if)) _ vol(pr(if p )) _ , 
vol(pr(K')) vol(pr(^)) lP + JU? J ' 

This finishes the proof of Theorem 9.2 □ 

Proof of Lemma 9.3, following Kottwitz [16]. We may replace G/Z and G'/Z' by 
their simply connected coverings G resp. G' and pr(K p ) and pr(K p ) by their 
inverse images _R" P resp. K' p . We use on G(Q P ) resp. G'(Q P ) the Haar measure 
induced by a top differential form on the Z p -form of G resp. G' corresponding 
to an Iwahori subgroup I p C K p resp. 7 p C K' p . These measures are compatible, 
cf. [16], p. 632. The volumes of I p and I' are related as follows. Choose as in 
[16] a maximal split torus S in G and a maximal torus S\ containing S which 
splits over an unramified extension. We also denote by S\ the canonical Z p -form 
of Si . Choose S", S[ of the same sort for G' . Then 

vol(4) _ Si(¥ p ) _ Qp-1) 2 

vol(J p ) 5i(F p ) p 2 -l ' 

since in the case at hand Si = and 5^ = ResQ p2 /Q p G m . The result follows 
since 

\K p /I p \ = l + 2p + 2p 2 + 2p 3 +p 4 , \K' p /I' p \=p+l , 

hence 

vol(pr(K p )) _ vol(/ p ) \K p /I p \ (p - l) 2 p 4 - 1 



vol(pr(Kp) vol(/ p ) |K p /J p | P 2 -l p 



□ 



We next formulate the corresponding result for the intersection of special cycles. 

For ni, . . . ,n r with 1 < rii < 4 and with ni + . . . + n r = 4 , let d\ G 
Sym ni (Z( p ))>o and fix locally centrally symmetric K p -invariant open compact 
subsets Ui cV(A p f ) ni . Let 

(9.3) W = Wi + . . . + W r 

be a decomposition of W into symplectic subspaces of dimensions 2ni , compatible 
with the fixed symplectic basis, and let 

(9.4) l : Hi A x ... x H rA — ► H A 

be the corresponding homomorphism of metaplectic groups, covering the embedding 

(9.5) t : Sp(W hA ) x ... x Sp(W rA ) Sp(Wa). 

Restricting to the archimedean place, for (/ii, . . . , /i n ) e -Hi,oo x . . . x if r)00 , we 
have 

(9.6) W* (ifa, • • • , M) = Wl (hi) . . . Wl(h r ), 
where T has diagonal blocks di,...,d r . Thus, by (7.3), we obtain: 
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Corollary 9.4. With the above notations, 



J2E'(i(h u ...,h r ),0,^) 



= -vo\(SO(V')(R)) ■ wlfa) . ..Wl(h r ) 



T 



x vol(pr(K)) ■ logp < Z(di, uji), . . . , Z(d r , uj r ) > 



proper 
V 



where the intersection number on the right side is defined by (7.3), and the sum- 
mation runs over T e Sum^iT,^)^ such that Diff(T,C) = {p} , diag(T) = 
(di,...,d r ), and T represents 1 over Z p . Also, $ is determined as in The- 
orem 9.2 with u> = u>i x • • • x u r . 

Of course, the left side of the expression of Corollary 9.4 is part of the {d\ , . . . , d r ) - 
th Fourier coefficient of the pullback 



(9.7) 



F(hi, . . .,h r ;G>) :=E'(t(h 1 ,...,h r ),0,^), 



cf. [19], (6.13). This result gives an analogue of the results of [19]. 
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§10. Representation densities. 

In this section, we give the proof of Propositions 9.1, which is based on a for- 
mula of Kitaoka, [14], for representation densities. We then describe a conjectural 
generalization of Kitaoka's formula. In this section, for x G Q p , x( x ) = ( x ,p)p ■ 

We begin by recalling the well known relation between the values of the function 
Wr, P (e 7 s, <E» P ) , at integer values of s and classical representation densities. 

For a suitable choice of basis for V(Z p ) the quadratic form q has matrix 
(10.1) S = So=( i-l 2 



For r > , let 



'S 

(10.2) S r = ( \ • l r 



1 • 1 

2 lr 



For nonsingular matrix T G Sym^iXp) , let 
(10.3) 

a p (S r ,T) = lim p-*( 10+8r )#{ x e M 5+ 2rA Z /P tz ) I " T e p*%m 4 (Z p ) } 



be the classical representation density [15], p. 98. This quantity depends only on 
the GL 4 (Z p ) -equivalence class of T , so we assume that 

(10.4) T = diag(e p ao , e lP a \ e 2 p a \ e 3 p a3 ), 

with €j G Z* and < ao < ai < a 2 < a 3 . Then, as explained in Corollary A. 1.5 
of [19], W TtP (e, r, %) = if T G Sym 4 (Q p ) \ Sym 4 (Z p ) , and 

(10.5) ^T, P (e,r,$ P ) = a P (S r ,T) 

if T G Sym 4 (Z p ) , since the factor 7 P (Vp) in loc.cit. is 1 in our present case. 
Recall - see [14], Lemma 9 and the discussion on pp. 450-453, for example - 
that a p (S r ,T) is a rational function of X = p~ r , i.e. there is a rational function 
As,t(X) such that 

(10.6) a p (S r ,T) = A s , T (p- r ) • 
We therefore have 

(10.7) W^ p (e,0,%) = -log(p) • ^{A s ,t(X)}\ x=1 . 
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At this point we have proved part (i) of Proposition 9.1. 

Similarly, let (p' p be the characteristic function of the lattice y'(Z p ) 4 = V^ p \Z p ) 4 
and let <fr p (s) be the corresponding standard section. Again, for a suitable choice 
of basis for V'(Z P ) , the quadratic form on V'(Z P ) has matrix 

(10.8) S' = S' = di ag (l,l,p,p,-pp), 
where f3 G Z p \ Z p ' 2 . Again, the factor 7 P (V p ') = 1 , and so 

(10.9) W T , p (e, 0, $y = p~ 4 ■ a p (S' , T). 

The following two results imply parts (ii) and (hi) of Proposition 9.1. 

Proposition 10.1. Suppose that T e Sym^Zp) is not represented by V(Q P ) and 
that T represents 1 . Let e p (T) be the local intersection multiplicity, given by the 
formulas of Proposition 6.1. Then, 



»A 

= logp-(l-p- i )(l-p-' 2 )-e p (T). 



W^ p (e,0,%) = -log(p) • -^{A S , T (X)} 



x=i 



Proposition 10.2. Suppose that T e Sym 4 (Z p ) with det(T) ^ represents 1 . 
Then 

„ r , _ / v _ 4 /c/ ^ ( P" 4 (l-p- 2 )2(p + l) */V'(Q p ) representsT 

otherwise. 



Of course, we would like to have analogous information about (e,0, <fr p ) 

and Wr iP (e, 0, <J> p ) for all T . At first, we simply restrict to the case where p \ T , 
so that, we may assume that ao = , i.e., 

(10.10) T = diag(e , e lP a \ e 2 p a \ e 3 p a3 ). 

Note that S ~ I5 . Then, by the standard reduction formula, [13], p. 149, 

(10.11) Oi P (S r , T) = a p (S r , e )a p (S r , T), 
where S r is obtained by adding a split space of dimension 2r to 

(10.12) 5 = diag(l,l,l,e ) 
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and 

(10.13) f = dmg(e lP a \e 2 p a \e 3 p a3 ). 
Note that 

(10.14) a p (S r , e ) = (1 + xMp" 2 ^) = (1 + x(e )p" 2 X), 
where X = p~ r , [32]. 

Now suppose that x(e ) = 1 , i.e., that T represents 1 . Let H 2m be the split 
quadratic form of rank 2m over Z p , so that 



(10.15) H 2m = 



i-ro 



Then S r is isomorphic to the split space H 2r+4 , and Kitaoka gives an explicit 
formula for the representation density ap(H 2 m,T) for any ternary form T, [14]. 
His formulas, in the cases a\ — a 2 even and cii — a 2 odd, are given as a sum of 
five double sums! These can be simplified to yield the following expressions: 

Proposition 10.3. (Kitaoka, [14]) Let X = p~ r , and let 

f = diag(e lP a \e 2 p a \e 3 p a3 ), 

with < ai < a 2 < a 3 . 
Let 

fl if cii = a 2 = CI3 mod (2), 

X(~eie 2 ) ifa x =a 2 ^a 3 mod (2), 
x(-eif3) ifa!^a 2 ^a 3 mod (2), 
, x(-e2C3) ifai^a 2 = a 3 mod (2). 
(i) If ai = a 2 mod (2) , then 

a p (H 2r+4 ,f) 



(l-p- 2 X)(l-p- 2 X 2 ) 

2 i , min(ai,£j . 

^2 p £ ( ^2 X 2t ~ k + x(T)X ai+a2+CL3+k ~ 2£ j 
t=o ^ k=o ' 

(dl \ / «3 — a 2 \ 

fc=0 ' ^ 7=0 ' 



where e = x(— €162) . 

(ii) If ai ^ a 2 mod (2) , t/ien 
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Qtp 



(l-p- 2 X)(l-p- 2 X 2 ) 



"1 +09 — 1 . , 

min(a 



— 2 . mimai,^ 

pi I .V 2 ' '•' ■ \(7').v" ; ' " 

^=0 ^ k=0 



Note that these expressions exhibit the functional equation of the local degener- 
ate Whittaker function under X 1— > X -1 . Evaluating at X = 1 and taking (10.11) 
and (10.14) into account, we obtain: 

Corollary 10.4. Suppose that T represents 1 . 
(i) If a\ = a 2 mod (2) , then 

a l+ a 2 1 



( l-°-ix'i r -F---) =(1 + X(f)) £ (mi " (0l '° + 1)P< 

j=o 7 



where e = x(— e^) ■ 

(ii) If a\ ^ a2 mod (2) , i/ien 



a l+ a 2 — 1 
2 



(i- P a - 2 Ki T -p- 4 ) =(1 + x(:f)) ^ (min( ai ,i) + iy. 



In case (ii), this quantity vanishes if and only if %(T) = — 1 . In case (i), if 
02 = 03 mod (2) , then x(T) = 1 and there are an odd number of terms in the last 
sum, so that the whole expression is nonzero. If 0,2 ^ as mod (2) , then %(T) = 
x(— e i e 2) = e , so that the whole expression vanishes if and only if %(T) = —1 . 

Proposition 10.5. Suppose that T represents 1 . Also suppose that x(T) = —1 , 
so that T is not represented by S , i.e., by V(Q P ) 
(i) If a,\ = a2 mod (2) , then 

d [ A s ,t(X) ) 

dX\(l- p~ 2 X 2 )(l - p-^X 2 ) J 

a l+ a 2 1 / A\ 

2 / mln (ll j ") \ 

P[ (a 1 + a 2 + a 3 + 2k-4£)j 



X=l 



1=0 x k=0 

-p * (ai + l)( o )• 



(ii) If ai ^ a 2 mod (2) , i/ien 
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ax {(l-p- 2 X)(l-p- 4 X 2 ' 



a 1 +a 2 -l . 

2 min(ai,t) 



- (ai + a 2 + a 3 + 2k-4£). 



After a short manipulation, these expressions coincide, up to sign, with those 
given in Proposition 6.1 for the local intersection multiplicity e p (T) ! 

Corollary 10.6. Suppose that p\T and eo is a square, i.e., that T represents 
1 over Z p . Also suppose that T is not represented by S . Then 



d -{A s ,r(X)} 



= -(l-p-')(l-p-*)e p (T), 

x=i 



dX 

where e p (T) is as in Proposition 6.1. 

This completes the proof of Proposition 10.1. 

Proof of Proposition 10.2. We apply the reduction formula to obtain: 

(10.15) a p (S' r ,T) = a p (S' r7 e )a p (S' rl T) 1 
where S r is obtained by adding a split space of dimension 2r to 

(10.16) S' = diag(l,e /3,p,-p/3) 
and T is as in (10.12). 

If eo is a square, then 

a p (^ / ,eo) = (l-x(-l)p- 1 ), 

[32]. On the other hand, S' is just the norm form on the maximal order of the 
division quaternion algebra over Q p . 

Lemma 10.7. 

a p (S\f)= 2(1 + X (-l)p- 1 )(p + I). 



Proof. Let B be the division quaternion algebra over Q p , and let R be its maximal 
order. Then, for a suitable 7L P -basis, 5" is the matrix for the quadratic form Q 
given by the reduced norm on R . Let 

A p r(T) = #{x G (R/p r R) 3 | Q[x] = f mod p r }, 
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so that 

r *. p \U ,-L ) — 11111 JJ js. v 



a p (S',f) = lim p- 6r A pr (T). 



Choose a uniformizer tc <E R such that n 2 = — p , and hence Q[ttx] = pQ[x] . Note 
that x G R if and only if Q[x] G Z p . Thus there is a bijection 

G {R/p r Rf | = mod p r } 

{y G {R/p^TxRf | = T mod p"" 1 }, 

given by x i— > 7r _1 x . Since |i?/7ri?| = p 2 , we have 

V(pT)=/V- 1 (T), 

and hence 

a p (S',pT) = a p (S',T). 

Thus, we may replace T by T" = diag(ei, e2f» a2_ai , e3p a3_ai ) . Here e\ can be 
taken to be equal to either 1 or (3 . Using reduction, we have 

a p (S', T') = a p (S', e 1 )a p (S" ', T"), 

where 

S" = diag(ei/3, p, -(3p) , and T" = diag(e 2 p a2 ~ ai , e 3 p aa ~ ai ) . 
By Theorem 3.1 of [32], 

a p (S',e 1 ) = (l + X (-l)p- 1 ). 

If ei = 1 , the form 5"' is just the norm form on the trace zero elements in 
R , while, if e± = (3 , then S" is isomorphic to (3 times this norm form. Since 
a p (0S" ', 0T") = a p (S",T") , Proposition 8.6 of [19] yields 



a p (b ,T 



2(p + 1) if T" is anisotropic, 
otherwise. 



□ 



Remark 10.8. The value a p (S', T) is given erroneously by Gross and Keating as 
2 • (1+p- 1 ) ■ (p + 1) , [7], Proposition 6.10. 



Thus 



a p (S\ T) = a p (S' 7 l)a p (S\ T) 
= 2(l-p- 2 )(p+l), 
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as claimed in Proposition 10.2. □ □ 

Notes on Clifford algebras 

A.l. Let (V,q) be a non-degenerate quadratic space of dimension 5 over a field 
F of characteristic not 2. Let C(V) be its Clifford algebra, with its 2-grading 

C(V) = C+(V)®C-(V) . 

The Clifford involution c i— > c' of C(V) is the unique involution which acts by 
the identity map on V C C~(V) . Thus 

(vi ■ ■ ■ v r )' = v' r ■ ■ ■ v[ . 

If v\ , . . . , i>5 is a basis for V , then the element 5 = v\ • • • v$ lies in the center of 
CiV) and satisfies 

S' = 6 . 

Let 

G = GSpin(F) = {ge C+(V) X \ gV g~ x = V, and gg' = u(g)} 
which may be considered as an algebraic group over Spec F . 

A. 2. In this section suppose that F is algebraically closed and choose a Witt 
decomposition of the quadratic space V , 

V = V + © V © V- 

where dim V± = 2 and V± are maximal isotropic subspaces of V . Let vq G Vq 
be a basis vector with q(v ) = 1 . We recall the Spin representation of G . We use 
the identifications of representations of C(V) , 

C(V)/C(V)C(V-) >0 = C(V+ © Vq) = 

= C(V+)(l + v )®C(V + )(l-v ) . 

As C(V) + -modules the last two modules are isomorphic. Either one of them 
defines the Spin representation W of G . Its dimension is 4. 

Fix an isomorphism A 2 V+ = F and let 

A : W -> F 

be the linear functional obtained by composing this isomorphism with the projection 
of C(V + ) = A(V+) onto A 2 V + . We obtain an alternating F-form on W by 

< x, y >= X(x'y) . 
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Lemma. For c G C(V) , and for x and y G W , 

< a(c)x, y >=< x, cr(c)y > . 

In particular, for g G G = GSpin(V) , 

< <r(g)x, a(g)y >= u(g) < x,y > . 

Here a(g) denotes the spin representation action of g on W , and v : G — > F x , 
v(g) = gg' is the restriction to G of the spinor norm on C(V) . 

Proof. Choose a basis bq, ei, vq, fo, fi for V such that the matrix for the 
quadratic form is 

/ 1 OX 

1 

1 

1 

Vo 1 / 

In C(V) , vl = 1 , e fo + foe = 1 , ei/i + /iei = 1 , e\ = , v (l+v ) = (l + v Q ) , 
etc. The spin representation W = C(V + )(1 + vo) has basis (1 + vo) , eo(l + fo) > 
eoei(l + vq) , and ei(l + vq) . We take A to be the coefficient of eoei(l + fo) and 
the symplectic form has matrix 




It is easy to check that 



o-(e ) 



<x(ei) 





/° 












1 






















1 




Vo 








0/ 




/0 








°\ 





















-1 










vl 








0/ 




1 








> \ 







-1 
















1 





V 











-1/ 




f° 


1 





°\ 
































Vo 





1 


0/ 
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and 



*(/0 



/O 1 
0-10 




\0 0/ 

If cr(c) is any of these matrices, then JV(c) J -1 = cr(c) , and hence, for any 
c G C(V) , JV(c) J" 1 = (t(c') , as claimed. □ 

Corollary, a : G = GSpin(V) GSp(W). 

A. 3. In this section F is again arbitrary, of characteristic not 2. 

Lemma. Let (V, q) be a non- degenerate quadratic space of dimension 5. The 
subspace 5 ■ V C C + (V) is characterized as: 



Proof. Recall that 6 G C~(V) is central in C(V) and satisfies 6' = 5 . It is, thus, 
clear that x = 5v satisfies x' = x . On the other hand, x 2 = q(v)5 2 = a lies in 
F , the center of C + (V) . In addition, if s^0, then x cannot lie in the center of 
C + (V) , since, if it did, then v = 5~ l x would lie in the center of C(V) , and this 
is not the case. If a = , so that x 2 = , the condition tr(x) = is immediate. 
If x 2 = a 7^ , choose u G V with q(u) ^ but with (u, v) = , and set y = 5u . 
Then xy = —yx , and so, over an algebraic closure of F , left multiplication by y 
gives an isomorphism between the ±y/a eigenspaces of x , and thus these spaces 
have the same dimension and tr(x) = . This proves that 5V is contained in 
the space on the right hand side. The converse inclusion will be proved further 
down. □ 

Let B be a quaternion algebra over F with main involution i , and let C = 
M 2 (B) with involution x i— > x' = t x b . Let 



S-V = {xe C+(V) \x' = x and tr(x) = }. 



Vb = { x G C | x' = x and tr(x) = } 




Note that 




so that the inclusion Vb ^ M 2 {B) induces a homomorphism 



C(V B ,q B ) 



M 2 {B), 
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where the quadratic form on V B is qB (x) = xx' . The diagram 

C(V B ,q B ) — > M 2 (B) 

C(V B ,q B ) — > M 2 (S) 

commutes, and induces an isomorphism C + (Vb, Qb) M 2 (B) , compatible with 
the involutions. 

Conversely, let V be a nondegenerate quadratic space of dimension 5. The 
Clifford involution induces an isomorphism C + (V) ~ C + (V) op , hence C + (V) is 
of the form 

C+(V)~M 2 (B) , 

for a quaternion algebra B over F . We may choose the isomorphism compatible 
with the involutions xhx'. This map then carries 5V into Vb ■ For dimension 
reasons we obtain an isometry, 

(V,5 2 -q v )^(V B ,q B ) . 

This also concludes the proof of the lemma above. 

Corollary. 

G = {geC + (Vr\gg' = u(g)} . 

A. 4. Any involution of the central simple algebra C = M n (B) , has the form 
x I— > hx'h~ l where x' = t x L , where h G GL n (B) with h! = ±h . If b! = h , we 
say that the involution is of mam type, while, if h! = — h , we say that h is of 
nebentype. As observed above, the Clifford involution on M 2 (B) is of main type. 

Let £ be a central simple algebra over F , with dim^ E = 16 2 , and with a 
nontrivial involution x ^ x v whose restriction to F is trivial. Then there is a 
quaternion algebra B over F and an isomorphism E ~ M 8 (B) . For a quaternion 
algebra _E?i over F, let Ci = M 2 (i?i) and let a; i— > x 771 be an involution of C\ 
whose restriction to F is trivial. Suppose that there is a (unitary) homomorphism 

h :Ci = M 2 (B 1 ) ^E = M 8 (B) 

such that 

n(cf = i 1 ( c " 1 )- 



Let 



C 2 = Cents (zi(Ci)) 
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be the centralizer of the image of C\ and let i 2 : C 2 -E 1 be the natural inclusion. 
Then C 2 ~ M 2 (B 2 ) , where 

£i<8>£ 2 ~M 2 (B), 

and we have an isomorphism 

i = ii ® i 2 : Ci ® C 2 £■ 

such that 

z(c 1 ®c 2 ) r ' = z(cf ®c^ 2 ), 

for an involution ?? 2 of C 2 . 

Proposition. TTie types of the involutions rj = r\\ <E> ?] 2 are; 

mam <g> neben 
neben <g> main 

main <8> main 
neben <g> neben. 

Proof. We can assume that F is algebraically closed. Then, on B = B\ = B 2 = 
M 2 (F) , 

/a oY _ / a 7 -o\ 
Vc dy) ~ ^-c a J' 

as usual, and a main involution on C\ = C 2 = M 2 (B) or on E = Mg(B) , is given 
by x 1— > . On M n (B) ~ M 2n (F) , this amounts to applying transpose on the 
matrix of the 2x2 blocks and then applying 1 blockwise. We denote this type of 
transpose by x i— > *x and write x x for the usual transpose on M 2n (F) . Let 

r = ^ 1 1 ^ G S = M 2 (F) , so that t l = -r , and, for x e B , 

TX L T~ l = T X. 

Setting 

h = h n = diag(r, . . . , r) 
in M n (B) , we have involutions of nebentype 

which, on M n (B) ~ M 2n (F) are just given by x 1— > T x , the itsitai transpose, rather 
than the blockwise transpose. 



mazn = 



and 

neben = 
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Now consider the explicit isomorphism 

i : M 4 (F) <g> M 2 {B) M 8 (B) 
(a-ij) (g)y^> (a^y). 

Applying the involution of main type on M$(B) , we have 

t i(x <g> y) L = i( T x <g> t y t ). 

Similarly, applying the involution of nebentype on Mg(B) , we have 

T i(x <g) y) = i( T x <g> tfy'h' 1 ) = i( T x ® T y). 

Every involution on E compatible with the isomorphism i : C\ ® C 2 ^> E 
is conjugate to one of these two by an element of the form g = i(gi <g) g%) , with 
V = ±g . Note that 



and 



l 9" = 9 



l 9 l = -9 



T 9i = 9i and f g L 2 = g 2 
T 9i = -9i and f g L 2 = -g 2 , 

T 9i = 9i and l g l 2 = -g 2 
T 9i = ~9i and t g L 2 = g 2 . 



Also observe that the involution 

x ^ gi^xgl 1 = gih t x L h~ 1 g^ 1 
is of main type if 7 g\ = —g\ and of nebentype if T g\ = g\ , since 
T 9l = h'g^- 1 = ± 9l ^ ^(gihY = - 9l h. 

□ 

A. 5. Let B be a quaternion algebra over M. . 

Lemma. For r G B x with t l = ±r , the involution x \— > x* = riV" 1 on B is 
positive if and only if: 

( t l = — t and r 2 < if B = M 2 (R) 

\ t l = t if B = H is division. 

In particular, if B = H , then x* = x L is the unique positive involution on B . 

Proof. Take r G B x such that r L = — r and r 2 < . Note that the condition 
on r 2 is automatic when B = H . Choose n G B x such that nr = —tv and 
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fj L = —r\ . Then every element x G B can be written uniquely in the form x = a+br] 
with a and b G K(r) ~ C . Then 

x* = r(a + briYr' 1 = a L - r] L b L 

and 

tr(xx*) = tr((a + bi])(a L - i] L b L )) 

= tr(aa L + br]a L - arfb 1 - br)rfb L ) 
= 2(aa L + bb L r] 2 ). 

If B = M2(M) , then rj 2 > , and this quantity is positive, while, if B = H , then 
i] 2 < , and this quantity can be negative. Note that, when B = M2(R) , then an 
involution defined by a r with r 2 > cannot be positive. □ 

A.6. Let B = M 2 (R) and let C = M 2 (B) with involution x' = V as above 
and let 

V = {x G C; x' = x and tr(x) = 0} . 
Then the signature of F for the form 55 of A. 3 is (3,2). 
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